requests源码框架浅析

Posted 2020-10-09 fuzzier

本文主要是对 requests 的 for human 结构的部分进行简单分析，对于里面具体的功能实现（比如cookies如何存储，http相关对接）没有深入研究。

1 对于requests主要模块说明：

1.1 __init__.py: 写入了requests的各种方法，可以直接调用

1.2 api.py: 定义了requests 的请求的各种方法（get, post, put, delete...)

1.3 session.py: 主要定义了Session类，里面封装这各种session的具体请求方法(Session.request()..)

1.4 model.py: 定义了各种模型（Request, Response)

1.5 utils.py: 定义了中途需要用的各种方法

 

2 requests的具体请求过程

html = requests.get(‘https://www.baidu.com‘, headers=headers).text

html2 = requests.Session().get(‘https://www.xxx.com‘, coookies=cookies).text

2.1 requests or session

不管是 requests.get 还是 requests.Session.get 他都会先去 调用调用 session.request(method=method, url=url, **kwargs) 方法

    with sessions.Session() as session:
        return session.request(method=method, url=url, **kwargs)

***此处补一句，  定义方法时候传入**kwargs这样类型的参数，很方便后面对象或方法接收参数。***

2.2 session.request 过程

所以现在我们来看 session.request 这个方法

    def request(self, method, url,
        params=None,
        data=None,
        headers=None,
        cookies=None,
        files=None,
        auth=None,
        timeout=None,
        allow_redirects=True,
        proxies=None,
        hooks=None,
        stream=None,
        verify=None,
        cert=None,
        json=None):
        """Constructs a :class:`Request <Request>`, prepares it and sends it.
        Returns :class:`Response <Response>` object.

..."""
        # Create the Request.
        req = Request(
            method = method.upper(),
            url = url,
            headers = headers,
            files = files,
            data = data or {},
            json = json,
            params = params or {},
            auth = auth,
            cookies = cookies,
            hooks = hooks,
        )
        prep = self.prepare_request(req)

        proxies = proxies or {}

        settings = self.merge_environment_settings(
            prep.url, proxies, stream, verify, cert
        )

        # Send the request.
        send_kwargs = {
            ‘timeout‘: timeout,
            ‘allow_redirects‘: allow_redirects,
        }
        send_kwargs.update(settings)
        resp = self.send(prep, **send_kwargs)

        return resp

2.3 Request类

首先将接收到的各种参数（headers, cookies,...）传入Request类进行封装

class Request(RequestHooksMixin):

    def __init__(self, method=None, url=None, headers=None, files=None,
        data=None, params=None, auth=None, cookies=None, hooks=None, json=None):

        # Default empty dicts for dict params.
        data = [] if data is None else data
        files = [] if files is None else files
        headers = {} if headers is None else headers
        params = {} if params is None else params
        hooks = {} if hooks is None else hooks

        self.hooks = default_hooks()
        for (k, v) in list(hooks.items()):
            self.register_hook(event=k, hook=v)

        self.method = method
        self.url = url
        self.headers = headers
        self.files = files
        self.data = data
        self.json = json
        self.params = params
        self.auth = auth
        self.cookies = cookies

    # ...

2.4 对request和传入的参数进行处理

然后将封装好的对象传入 prepare_request(self, request) 方法

    def prepare_request(self, request):

        cookies = request.cookies or {}

        # Bootstrap CookieJar.
        if not isinstance(cookies, cookielib.CookieJar):
            cookies = cookiejar_from_dict(cookies)

        # Merge with session cookies
        merged_cookies = merge_cookies(
            merge_cookies(RequestsCookieJar(), self.cookies), cookies)

        # Set environment‘s basic authentication if not explicitly set.
        auth = request.auth
        if self.trust_env and not auth and not self.auth:
            auth = get_netrc_auth(request.url)

        p = PreparedRequest()
        p.prepare(
            method=request.method.upper(),
            url=request.url,
            files=request.files,
            data=request.data,
            json=request.json,
            headers=merge_setting(request.headers, self.headers, dict_class=CaseInsensitiveDict),
            params=merge_setting(request.params, self.params),
            auth=merge_setting(auth, self.auth),
            cookies=merged_cookies,
            hooks=merge_hooks(request.hooks, self.hooks),
        )
        return p

该方法主要是对传入个各种参数（headers, cookies,...）进行相应的处理以便后续调用，

这里他先将cookies进行了处理，然后对所有传入的参数进行相应处理，调用PrepareRequest.prepare 方法进行处理

    def prepare(self, method=None, url=None, headers=None, files=None,
        data=None, params=None, auth=None, cookies=None, hooks=None, json=None):
        """Prepares the entire request with the given parameters."""

        self.prepare_method(method)
        self.prepare_url(url, params)
        self.prepare_headers(headers)
        self.prepare_cookies(cookies)
        self.prepare_body(data, files, json)
        self.prepare_auth(auth, url)

        # Note that prepare_auth must be last to enable authentication schemes
        # such as OAuth to work on a fully prepared request.

        # This MUST go after prepare_auth. Authenticators could add a hook
        self.prepare_hooks(hooks)

可以看出，每个具体的请求参数都有对应的参数进行处理。

2.5 发送得到response

request请求对象设计好后，send(request)，得到 response