python asyncio 异步 I/O - 实现并发http请求(asyncio + aiohttp)

Posted 2022-03-08 上海-悠悠

前言

如果需要并发 http 请求怎么办呢？requests库是同步阻塞的，必须等到结果才会发第二个请求，这里需使用http请求异步库 aiohttp。

环境准备

aiohttp 用于 asyncio 和 Python 的异步 HTTP 客户端/服务器。
使用pip安装对应的包。当前使用版本v3.8.1

pip install aiohttp 

并发http请求

如果使用requests 库，发10个请求访问我的博客，那么这10个请求是串行的。

import requests
import time

url = "https://www.cnblogs.com/yoyoketang/"

start_time = time.time()
for i in range(10):
    r = requests.get(url)
    print(r)
print('总耗时：', time.time()-start_time)

运行结果,总共耗时1秒左右

<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
<Response [200]>
总耗时： 1.0870192050933838

使用asyncio + aiohttp 并发请求

import asyncio
from aiohttp import ClientSession
import time


async def bai_du(url):
    print(f'启动时间: time.time()')
    async with ClientSession() as session:
        async with session.get(url) as response:
            res = await response.text()
            return res


async def main():
    url = "https://www.cnblogs.com/yoyoketang/"
    task_list = []
    for i in range(10):
        task = asyncio.create_task(bai_du(url))
        task_list.append(task)
    done, pending = await asyncio.wait(task_list, timeout=None)
    # 得到执行结果
    for done_task in done:
        print(f"time.time() 得到执行结果 done_task.result()")

# asyncio.run(main())
start_time = time.time()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
print("总耗时: ", time.time()-start_time)

运行结果

启动时间: 1646028822.3952098
启动时间: 1646028822.4161537
启动时间: 1646028822.4171515
启动时间: 1646028822.4171515
启动时间: 1646028822.4171515
启动时间: 1646028822.4171515
启动时间: 1646028822.4181483
启动时间: 1646028822.4181483
启动时间: 1646028822.4181483
启动时间: 1646028822.4181483
....
总耗时:  0.17400002479553223

从运行结果可以看到，启动时间非常接近，也就是并发的请求，总过耗时 0.17 秒。

asyncio.run

需注意的是这里使用 asyncio.run(main()) 会报错RuntimeError: Event loop is closed

Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x00000238675A8F70>
Traceback (most recent call last):
  File "D:\\python3.8\\lib\\asyncio\\proactor_events.py", line 116, in __del__
    self.close()
  File "D:\\python3.8\\lib\\asyncio\\proactor_events.py", line 108, in close
    self._loop.call_soon(self._call_connection_lost, None)
  File "D:\\python3.8\\lib\\asyncio\\base_events.py", line 719, in call_soon
    self._check_closed()
  File "D:\\python3.8\\lib\\asyncio\\base_events.py", line 508, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

解决办法，把执行方式 asyncio.run(main())改成

# asyncio.run(main())
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

注意原因是asyncio.run()会自动关闭循环,并且调用_ProactorBasePipeTransport.__del__报错, 而asyncio.run_until_complete()不会.
详情参考https://zhuanlan.zhihu.com/p/365815189