aiohttp asyncio.TimeoutError 从无使用 ClientSession

Posted 2023-03-30

【中文标题】aiohttp asyncio.TimeoutError 从无使用 ClientSession

【英文标题】：aiohttp asyncio.TimeoutError from None using ClientSession

【发布时间】：2019-02-05 19:57:08

【问题描述】：

这是一个奇怪的错误，因为当我尝试/捕捉它时，它什么也没打印。

我正在使用 sanic 服务器来异步。同时收集一堆图像，超过 3000 张图像。

在处理较小的样本量时，我没有收到此错误。

简化示例：

from sanic import Sanic
from sanic import response
from aiohttp import ClientSession

from asyncio import gather

app = Sanic()

@app.listener('before_server_start')
async def init(app, loop):
    app.session = ClientSession(loop=loop)

@app.route('/test')
async def test(request):
    data_tasks = []
    #The error only happened when a large amount of images were used
    for imageURL in request.json['images']:
        data_tasks.append(getRaw(imageURL))
    await gather(*data_tasks)
    return response.text('done')

async def getRaw(url):
    async with app.session.get(url) as resp:
        return await resp.read()

这个错误可能是什么？如果这是我的主机/互联网的某种限制，我该如何避免？

如果有帮助，我正在使用 DigitalOcean 的基本液滴，配备 1vCPU 和 1GB RAM

全栈错误：

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/sanic/app.py", line 750, in handle_request
    response = await response
  File "server-sanic.py", line 53, in xlsx
    await gather(*data_tasks)
  File "/usr/lib/python3.5/asyncio/futures.py", line 361, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/lib/python3.5/asyncio/tasks.py", line 296, in _wakeup
    future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 241, in _step
    result = coro.throw(exc)
  File "server-sanic.py", line 102, in add_data_to_sheet
    await add_img_to_sheet(sheet, rowIndex, colIndex, val)
  File "server-sanic.py", line 114, in add_img_to_sheet
    image_data = BytesIO(await getRaw(imgUrl))
  File "server-sanic.py", line 138, in getRaw
    async with app.session.get(url) as resp:
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 690, in __aenter__
    self._resp = yield from self._coro
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 277, in _request
    yield from resp.start(conn, read_until_eof)
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client_reqrep.py", line 637, in start
    self._continue = None
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/helpers.py", line 732, in __exit__
    raise asyncio.TimeoutError from None
concurrent.futures._base.TimeoutError

【问题讨论】：

您的任务正在消耗您的带宽。

@yorodm 我想这很难通过 python 来限制，我应该限制最大并发连接数吗？

你能检查一下here 的答案，看看它们是否适合你吗？

@yorodm 我认为this 可能就足够了吗？限制 aiohttp 中的池大小

【参考方案1】：

一次启动一百万个请求没有任何好处。将其限制为 10 或任何有效的值，并在继续循环之前等待它们。

for imageURL in request.json['images']:
    data_tasks.append(getRaw(imageURL))
    if len(data_tasks) > 10:
        await gather(*data_tasks)
        data_tasks = []        
await gather(*data_tasks)

【讨论】：

确实有道理！你认为来自 aiohttp 的这个选项就足够了吗？ docs.aiohttp.org/en/stable/…