连接没有通过 Python3 异步并发 HTTP 获取请求关闭

Posted 2023-03-30

【中文标题】连接没有通过 Python3 异步并发 HTTP 获取请求关闭

【英文标题】：Connections aren't closing with Python3 asyncio concurrent HTTP get requests

【发布时间】：2014-06-19 03:01:08

【问题描述】：

我刚刚开始使用 Python3.4 中的 asyncio 库并编写了一个小程序，它尝试一次同时获取 50 个网页。该程序在数百个请求后因“打开的文件过多”异常而崩溃。

我认为我的 fetch 方法通过 'response.read_and_close()' 方法调用关闭了连接。

有什么想法吗？我是否以正确的方式解决这个问题？

import asyncio
import aiohttp

@asyncio.coroutine
def fetch(url):
    response = yield from aiohttp.request('GET', url)
    response = yield from response.read_and_close()
    return response.decode('utf-8')

@asyncio.coroutine
def print_page(url):
    page = yield from fetch(url)
    # print(page)

@asyncio.coroutine
def process_batch_of_urls(round, urls):
  print("Round starting: %d" % round)
  coros = []
  for url in urls:
      coros.append(asyncio.Task(print_page(url)))
  yield from asyncio.gather(*coros)
  print("Round finished: %d" % round)

@asyncio.coroutine
def process_all():
  api_url = 'https://google.com'
  for i in range(10):
    urls = []
    for url in range(50):
      urls.append(api_url)
    yield from process_batch_of_urls(i, urls)


loop = asyncio.get_event_loop()
loop.run_until_complete(process_all())

我得到的错误是：

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/aiohttp/client.py", line 106, in request
  File "/usr/local/lib/python3.4/site-packages/aiohttp/connector.py", line 135, in connect
  File "/usr/local/lib/python3.4/site-packages/aiohttp/connector.py", line 242, in _create_connection
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/asyncio/base_events.py", line 424, in create_connection
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/asyncio/base_events.py", line 392, in create_connection
  File "/usr/local/Cellar/python3/3.4.1/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socket.py", line 123, in __init__
OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

【问题讨论】：

这对我来说很好用。您使用的是哪个版本的aiohttp？我有 0.8.1。

【参考方案1】：

啊哈，我想你的问题。

显式连接器绝对可以解决问题。

https://github.com/KeepSafe/aiohttp/pull/79 也应该为隐式连接器修复它。

非常感谢您发现 aiohttp

中的资源泄漏

UPD。 aiohttp 0.8.2 没有问题。

【参考方案2】：

好吧，我终于让它工作了。

原来我必须使用一个 TCPConnector 来汇集连接。

所以我做了这个变量：

connector = aiohttp.TCPConnector(share_cookies=True, loop=loop)

并将其传递给每个获取请求。我的新 fetch 例程如下所示：

@asyncio.coroutine
def fetch(url):
  data = ""
  try:
    yield from asyncio.sleep(1)
    response = yield from aiohttp.request('GET', url, connector=connector)
  except Exception as exc:
      print('...', url, 'has error', repr(str(exc)))
  else:
      data = (yield from response.read()).decode('utf-8', 'replace')
      response.close()

  return data