使用 urllib2、socks5 代理和 socksipy 时超时不起作用

Posted

技术标签:

【中文标题】使用 urllib2、socks5 代理和 socksipy 时超时不起作用【英文标题】:Timeout not working using urllib2, socks5 proxy and socksipy 【发布时间】:2012-07-04 18:59:38 【问题描述】:

我在 Python 2.6 中使用 socksipy 和 urllib2。一切正常,除了当我点击一个挂起的 URL 时超时。 urllib2 函数超时参数或全局套接字默认超时都不起作用。我什至尝试在下面的子类处理程序中以多种不同的方式设置超时,但没有成功。有什么想法吗?

这是一个测试脚本(假设您已经安装了 socksipy 项目并将其添加到您的系统路径中):

import os, sys
import httplib
sys.path.append( "/parent/path/to/socksipy/project" )
import socks # import socksipy
import socket
socket.setdefaulttimeout(30.0)
import urllib2

class SocksiPyConnection(httplib.HTTPConnection):
    def __init__(self, proxytype, proxyaddr, proxyport = None, rdns = False, username = None, password = None, *args, **kwargs):
        self.proxyargs = (proxytype, proxyaddr, proxyport, rdns, username, password)
        httplib.HTTPConnection.__init__(self, *args, **kwargs)

    def connect(self):
        self.sock = socks.socksocket()
        self.sock.setproxy(*self.proxyargs) 
        if isinstance(self.timeout, float):
            self.sock.settimeout(self.timeout)
        self.sock.connect((self.host, self.port))

class SocksiPyHandler(urllib2.HTTPHandler):
    def __init__(self, *args, **kwargs):
        self.args = args
        self.kw = kwargs
        urllib2.HTTPHandler.__init__(self)

    def http_open(self, req):
        def build(host, port=None, strict=None, timeout=0):
            conn = SocksiPyConnection(*self.args, host=host, port=port, strict=strict, timeout=timeout, **self.kw)
            return conn
        return self.do_open(build, req)

if __name__ == '__main__':

    #
    # this one works for non-hanging URL
    #
    proxyhost = "responder.w2"
    proxyport = 1050
    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, proxyhost, int(proxyport))
    socket.socket = socks.socksocket
    resp = urllib2.urlopen("http://www.google.com", timeout=30.0)
    # hang here
    print "returned 1"


    #
    # one way to go about it for a hanging URL
    #
    proxyhost = "responder.w2"
    proxyport = 1050
    socks.setdefaultproxy(socks.PROXY_TYPE_SOCKS5, proxyhost, int(proxyport))
    socket.socket = socks.socksocket
    resp = urllib2.urlopen("http://erma.orr.noaa.gov/cgi-bin/mapserver/charts?version=1.1.1&service=wms&request=GetCapabilities", timeout=30.0)
    # it hangs here
    print "returned 2"


    #  
    # another way to go about it for hanging URL
    #
    proxyhost = "responder.w2"
    proxyport = 1050
    opener = urllib2.build_opener(SocksiPyHandler(socks.PROXY_TYPE_SOCKS5, proxyhost, int(proxyport)) )
    resp = opener.open("http://erma.orr.noaa.gov/cgi-bin/mapserver/charts?version=1.1.1&service=wms&request=GetCapabilities", timeout=30.0)
    # it hangs here
    print "returned 3"

【问题讨论】:

我正在尝试做这个人正在做的事情,但我需要超时:***.com/questions/2317849/… 使用请求模块会有帮助吗?这要简单得多。 docs.python-requests.org/en/latest/user/quickstart/#timeouts @hughdbrown:如果支持 socks5 代理,我可能会使用 requests 模块重写所有内容 我运行此代码并在 30 秒后在第二个 URL 上收到超时异常: urllib2.URLError: 也许这个答案可以帮助你:***.com/questions/8464391/… 【参考方案1】:

这对我有用:

socks.socket.setdefaulttimeout(7)

你应该避免直接编辑 python socks 库。

【讨论】:

【参考方案2】:

原来我上面提到的“挂起/超时”问题实际上是sockssipy socks.py 代码中的“阻塞”问题。如果您访问的端点仍然以 200 响应但不发送数据(0 字节),那么 socks.py 将阻塞,因为这就是它的写入方式。以下是创建自己的超时之前和之后:

socks.py 之前

def __recvall(self, bytes):
    """__recvall(bytes) -> data
    Receive EXACTLY the number of bytes requested from the socket.
    Blocks until the required number of bytes have been received.
    """
    data = ""
    while len(data) < bytes:
       data = data + self.recv(bytes-len(data))
    return data

socks.py 后超时

def __recvall(self, bytes):
    """__recvall(bytes) -> data
    Receive EXACTLY the number of bytes requested from the socket.
    Blocks until the required number of bytes have been received.
    """
    data = self.recv(bytes, socket.MSG_WAITALL)
    if type(data) not in (str, unicode) or len(data) != bytes:
        raise socket.timeout('timeout')
    return data

【讨论】:

以上是关于使用 urllib2、socks5 代理和 socksipy 时超时不起作用的主要内容,如果未能解决你的问题,请参考以下文章

基于go手动写个转发代理服务

怎么开socks5代理,!

推送:啥是socks5代理ip

如何正确认识socks5代理ip

怎样设置socks代理

SOCKS5代理的介绍