requests源码分析

Posted 2020-08-27 李雷雷

0.前言

(1) 拆部分reques中感兴趣t的轮子

(2)对一些感兴趣的pythonic写法做一些归纳

1.用object.__setattr__来初始化构造函数

反正我之前就是直接实例对象时把所有参数传入构造函数的,一般人都这样..但事实证明这种方式并不好(可能),所以后来作者又把这种方式改掉了...但原谅我也不知道这两者有什么好坏之分..

class Request(object):
    """The :class:`Request` object. It carries out all functionality of
    Requests. Recommended interface is with the Requests functions.
    
    """
    
    _METHODS = (‘GET‘, ‘HEAD‘, ‘PUT‘, ‘POST‘, ‘DELETE‘)
    
    def __init__(self):
        self.url = None
        self.headers = dict()
        self.method = None
        self.params = {}
        self.data = {}
        self.response = Response()
        self.auth = None
        self.sent = False
        
    def __repr__(self):
        try:
            repr = ‘<Request [%s]>‘ % (self.method)
        except:
            repr = ‘<Request object>‘
        return repr
    
    def __setattr__(self, name, value):
        if (name == ‘method‘) and (value):
            if not value in self._METHODS:
                raise InvalidMethod()
        
        object.__setattr__(self, name, value)

初始化操作:

def get(url, params={}, headers={}, auth=None):
    """Sends a GET request. Returns :class:`Response` object.
    :param url: URL for the new :class:`Request` object.
    :param params: (optional) Dictionary of GET Parameters to send with the :class:`Request`.
    :param headers: (optional) Dictionary of HTTP Headers to sent with the :class:`Request`.
    :param auth: (optional) AuthObject to enable Basic HTTP Auth.
    """
    
    r = Request()
    
    r.method = ‘GET‘
    r.url = url
    r.params = params
    r.headers = headers
    r.auth = _detect_auth(url, auth)
    
    r.send()
    
    return r.response

2.大量复杂的参数传递时采用**kwargs

用**kwargs可在方法间的传递大量参数，不需要自己每次都初始化一个dict用来传参(嗯,之前我就是这样的傻逼)

def get(url, params={}, headers={}, cookies=None, auth=None):
    return request(‘GET‘, url, params=params, headers=headers, cookiejar=cookies, auth=auth)

def request(method, url, **kwargs):
    data = kwargs.pop(‘data‘, dict()) or kwargs.pop(‘params‘, dict())

    r = Request(method=method, url=url, data=data, headers=kwargs.pop(‘headers‘, {}),
                cookiejar=kwargs.pop(‘cookies‘, None), files=kwargs.pop(‘files‘, None),
                auth=kwargs.pop(‘auth‘, auth_manager.get_auth(url)))
    r.send()

    return r.response

3.monkey patch

热修复技术方案，可以参考协程,协程为了实现异步效果,替换了python原生的很多库。就是模块在加载前，把自己的模块在系统加载前替换掉原系统模块，然后达到自己的（不可告人的）目的。

这里其实不是requests使用了monkey patch,而是pyopenssl这个库,这个是为了修复python2.7中SNI的bug,将原来的ssl_wrap_socket方法做了替换(不过我没看到requests有任何注入操作,坑爹...)

# 替换
def inject_into_urllib3():
    ‘Monkey-patch urllib3 with PyOpenSSL-backed SSL-support.‘

    connection.ssl_wrap_socket = ssl_wrap_socket
    util.HAS_SNI = HAS_SNI
    util.IS_PYOPENSSL = True

# 还原
def extract_from_urllib3():
    ‘Undo monkey-patching by :func:`inject_into_urllib3`.‘

    connection.ssl_wrap_socket = orig_connection_ssl_wrap_socket
    util.HAS_SNI = orig_util_HAS_SNI
    util.IS_PYOPENSSL = False

如果在请求https过程中出现SNIMissing的问题,可以考虑这么解决:

pip install pyopenssl ndg-httpsclient pyasn1

try:
    import urllib3.contrib.pyopenssl
    urllib3.contrib.pyopenssl.inject_into_urllib3()
except ImportError:
    pass

相当于就是执行主动注入的操作(但这个不应该是requests框架自己该集成的么...)

4.hook函数

requests中有一个钩子函数，看历史版本，原来提供的回调入口有好几个，目前只有response一个回调入口了，测试代码如下

import requests

def print_url(r, *args, **kwargs):
    print r.content
    print r.url

requests.get(‘http://httpbin.org‘, hooks=dict(response=print_url))

这会发生什么呢？requests会在requests.Response返回前回调这个print_url这个方法。可以看到，回调操作是在requests拿到请求结果后才去操作的

    def send(self, request, **kwargs):
        """
        Send a given PreparedRequest.

        :rtype: requests.Response
        """
        ...

        # Get the appropriate adapter to use
        adapter = self.get_adapter(url=request.url)

        # Start time (approximately) of the request
        start = datetime.utcnow()

        # Send the request
        r = adapter.send(request, **kwargs)

        # Total elapsed time of the request (approximately)
        r.elapsed = datetime.utcnow() - start

        # Response manipulation hooks
        r = dispatch_hook(‘response‘, hooks, r, **kwargs)    

 

那dispatch_hook又干了什么呢？

def dispatch_hook(key, hooks, hook_data, **kwargs):
    """Dispatches a hook dictionary on a given piece of data."""
    hooks = hooks or dict()
    hooks = hooks.get(key)
    if hooks:
        if hasattr(hooks, ‘__call__‘):
            hooks = [hooks]
        for hook in hooks:
            _hook_data = hook(hook_data, **kwargs)
            if _hook_data is not None:
                hook_data = _hook_data
    return hook_data

可以看到dispatch_hook本身是可以拓展的，但可惜的是目前requests只有response入口了，也许是为了安全吧。

其实说真的,requests的hook使用起来真的不够好,真正好用的hook,可以看看flask.

5.上下文管理器（历史版本）

with requests.settings(timeout=0.5):
    requests.get(‘http://example.org‘)
    requests.get(‘http://example.org‘, timeout=10)

在with之中，所有的配置加载都是在局部生效的，就算requests.get(‘http://example.org‘, timeout=10)，但requests对象中的timeout属性依然是0.5而不是10，怎么实现的呢？

class settings:
    """Context manager for settings."""
    
    cache = {}
    
    def __init__(self, timeout):
        self.module = inspect.getmodule(self)
        
        # Cache settings
        self.cache[‘timeout‘] = self.module.timeout
        
        self.module.timeout = timeout
        
    def __enter__(self):
        pass
        
    def __exit__(self, type, value, traceback):
        # Restore settings 
        for key in self.cache:
            setattr(self.module, key, self.cache[key])

其实很简单，只要在进入这个context时，将原有的属性储存起来，退出context时，重新set回去就行了。

6.重定向redirect

requests对每一个send请求都会做重定向的判断,具体就是如果是重定向,那就执行以下这个方法

    def resolve_redirects(self, resp, req, stream=False, timeout=None,
                          verify=True, cert=None, proxies=None, **adapter_kwargs):
        """Receives a Response. Returns a generator of Responses."""

        i = 0
        hist = [] # keep track of history

        while resp.is_redirect:
            prepared_request = req.copy()

            if i > 0:
                # Update history and keep track of redirects.
                hist.append(resp)
                new_hist = list(hist)
                resp.history = new_hist
　　　　　　　...

            url = resp.headers[‘location‘]

            # Handle redirection without scheme (see: RFC 1808 Section 4)
            if url.startswith(‘//‘):
                parsed_rurl = urlparse(resp.url)
                url = ‘%s:%s‘ % (parsed_rurl.scheme, url)

　　　　　　　...
            extract_cookies_to_jar(prepared_request._cookies, req, resp.raw)
            prepared_request._cookies.update(self.cookies)
            prepared_request.prepare_cookies(prepared_request._cookies)

            # Rebuild auth and proxy information.
            proxies = self.rebuild_proxies(prepared_request, proxies)
            self.rebuild_auth(prepared_request, resp)

            # Override the original request.
            req = prepared_request

            resp = self.send(
                req,
                stream=stream,
                timeout=timeout,
                verify=verify,
                cert=cert,
                proxies=proxies,
                allow_redirects=False,
                **adapter_kwargs
            )

            extract_cookies_to_jar(self.cookies, prepared_request, resp.raw)

            i += 1
            yield resp

可以看到,requests会从url = resp.headers[‘location‘]取出重定向后的url,将resp追加到history中,然后重设head,cookie,proxy,auth执行self.send操作,然后yield resp后进入下一次循环,判断是否是redirect,最多redirect次数为30次.