grequests模块

Posted 2020-11-28 tulintao

grequests实际上就是封装了gevent里面的方法，然后配合上requests模块实现了异步的IO

grequests = gevent + requests + greenlet　　

grequests.map()内部的实现

def map(requests, stream=False, size=None, exception_handler=None, gtimeout=None):
    """Concurrently converts a list of Requests to Responses.

    :param requests: a collection of Request objects.
    :param stream: If True, the content will not be downloaded immediately.
    :param size: Specifies the number of requests to make at a time. If None, no throttling occurs.
    :param exception_handler: Callback function, called when exception occured. Params: Request, Exception
    :param gtimeout: Gevent joinall timeout in seconds. (Note: unrelated to requests timeout)
    """

    requests = list(requests)

    pool = Pool(size) if size else None
    jobs = [send(r, pool, stream=stream) for r in requests]
    gevent.joinall(jobs, timeout=gtimeout)

    ret = []

    for request in requests:
        if request.response is not None:
            ret.append(request.response)
        elif exception_handler and hasattr(request, ‘exception‘):
            ret.append(exception_handler(request, request.exception))
        else:
            ret.append(None)

    return ret

 

内部其实就是循环请求列表发起的请求，然后对返回的结果进行判断，然后添加到ret列表中，最终return ret就是最后我们看到的一个可迭代对象了。

>>> def exception_handler(request, exception):
...    print "Request failed"

>>> reqs = [
...    grequests.get(‘http://httpbin.org/delay/1‘, timeout=0.001),
...    grequests.get(‘http://fakedomain/‘),
...    grequests.get(‘http://httpbin.org/status/500‘)]
>>> grequests.map(reqs, exception_handler=exception_handler)
Request failed
Request failed
[None, None, <Response [500]>]