弹性 beantalk 中的 celery worker 出错（使用 django 和 SQS）[ImportError：curl 客户端需要 pycurl 库。]

Posted 2023-02-24

【中文标题】弹性 beantalk 中的 celery worker 出错（使用 django 和 SQS）[ImportError：curl 客户端需要 pycurl 库。]

【英文标题】：Error with celery worker in elastic beanstalk (using django and SQS) [ImportError: The curl client requires the pycurl library.]

【发布时间】：2019-06-20 13:36:16

【问题描述】：

我正在尝试使用 SQS 将一个使用 celery 定期任务的 django 项目部署到弹性 beanstalk。

我或多或少地遵循了这里的说明：

How to run a celery worker with Django app scalable by AWS Elastic Beanstalk?

当我部署到 eb 时，没有执行周期性任务。

检查 celery-beat 日志，一切正常：

celery beat v4.2.1 (windowlicker) is starting.
__    -    ... __   -        _
LocalTime -> 2019-01-27 09:48:16
Configuration ->
    . broker -> sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-01-27 09:48:44,659: INFO/MainProcess] beat: Starting...
[2019-01-27 09:48:44,660: INFO/MainProcess] Writing entries...
[2019-01-27 09:48:44,809: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2019-01-27 09:48:44,809: INFO/MainProcess] Writing entries...
[2019-01-27 09:48:49,865: INFO/MainProcess] Writing entries...
[2019-01-27 09:49:00,409: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:50:00,050: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:51:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:51:50,543: INFO/MainProcess] Writing entries...
[2019-01-27 09:52:00,048: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)
[2019-01-27 09:53:00,045: INFO/MainProcess] Scheduler: Sending due task sum_two_numbers (sum_two_numbers)

但芹菜工人似乎有问题。这是我在 celery-worker.log 中遇到的错误：

/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-01-27 09:49:12,513: INFO/MainProcess] Connected to sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
[2019-01-27 09:49:12,950: WARNING/MainProcess] /opt/python/run/venv/local/lib/python3.6/site-packages/celery/fixups/django.py:200: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-01-27 09:49:12,951: INFO/MainProcess] celery@ip-172-31-23-20 ready.
[2019-01-27 09:49:12,952: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/worker.py", line 205, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 317, in start
    blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/consumer/consumer.py", line 593, in start
    c.loop(*c.loop_args())
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/loops.py", line 91, in asynloop
    next(loop)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/hub.py", line 291, in create_loop
    item()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 163, in __call__
    return self.throw()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/vine/promises.py", line 160, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 316, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 332, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 342, in _get_async
    qname, count=count, connection=self.asynsqs,
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/transport/SQS.py", line 436, in asynsqs
    region=self.region
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/sqs/connection.py", line 27, in __init__
    **kwargs
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 178, in __init__
    **http_client_params)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/aws/connection.py", line 151, in __init__
    self._httpclient = http_client or get_client()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/asynchronous/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

 -------------- celery@ip-172-31-23-20 v4.2.1 (windowlicker)
---- **** -----
--- * ***  * -- Linux-4.14.88-72.76.amzn1.x86_64-x86_64-with-glibc2.3.4 2019-01-27 09:49:12
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         raiseflags:0x7fbf06309f28
- ** ---------- .> transport:   sqs://AKIAIVCNK32ABCHNNZSQ:**@localhost//
- ** ---------- .> results:
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . check_users_inboxes
  . periodic_send
  . raiseflags.celery.debug_task
  . scrapeURLs
  . spreadsheets_update
  . sum_two_numbers

我一直在寻找答案，有些人似乎也遇到了同样的问题。我尝试部署不同版本的 celery 和 kombu，但无法解决。

这是我的 requirements.txt 文件：

amqp==2.4.0
anyjson==0.3.3
beautifulsoup4==4.7.1
billiard==3.5.0.5
boto==2.49.0
bs4==0.0.1
cachetools==3.0.0
celery==4.2.1
certifi==2018.11.29
chardet==3.0.4
Django==2.1.5
django-celery-beat==1.4.0
django-celery-results==1.0.4
django-timezone-field==3.0
google-api-python-client==1.7.7
google-auth==1.6.2
google-auth-httplib2==0.0.3
httplib2==0.12.0
idna==2.8
kombu==4.2.2.post1
oauth2client==4.1.3
psycopg2==2.7.7
pyasn1==0.4.5
pyasn1-modules==0.2.3
python-dateutil==2.7.5
pycurl==7.43.0 --global-option="--with-nss"
python-crontab==2.3.6
pytz==2018.9
requests==2.21.0
rsa==4.0
six==1.12.0
soupsieve==1.7.3
stripe==2.18.1
uritemplate==3.0.0
urllib3==1.24.1
vine==1.2.0

如果我通过 ssh 连接到实例并键入“python -c 'import pycurl'”，一切似乎都正常，没有问题。

有人知道如何解决吗？任何帮助将不胜感激。

非常感谢。

【参考方案1】：

当你这样做时

python -c 'import pycurl'

确保python版本与python3.6相同， 当然，请尝试：

python3.6 -m pip install pycurl

【讨论】：

正如 holalluis 所建议的，问题在于该软件包被安装在 python2.7 而不是 python3.6 中。最重要的是，必须指定 --global-option="--with-nss" 才能找到包。通过执行export PYCURL_SSL_LIBRARY=nss 解决，然后执行python3.6 -m pip install pycurl --global-option="--with-nss" --target=/opt/python/run/venv/lib/python3.6/site-packages --no-cache-dir --upgrade

@jaume - 你是如何获得运行这些命令的权限的？我尝试使用 eb ssh 然后运行该命令，但没有运气。然后我尝试了sudo -i 并重新运行了您的命令。这似乎成功了，但我仍然从 venv python3.6 安装中得到一个错误，说明 libcurl 链接时后端与编译时不同。谢谢！

嘿@vitale232。我按照你建议的方式做了。一切正常，但后来我删除了所有环境。几个月后，当我尝试重新创建所有内容时，我得到了你说的同样的错误。所以我错了：这不是我解决它的方法。而且我不知道我是怎么解决的！！！我发布了另一个问题，但没有得到任何答案：***.com/questions/56043178/…

:-( 感谢您的回复。如果我弄清楚了，我会回答您的新问题。令我惊讶的是，在 eb 中配置所有内容是多么容易......除了 Celery。

我最终使用了 OpenSSL，它可以工作。一开始没有，但是当我改变kombu和celery的版本时，它起作用了。所以，在 requirements.txt 中： pycurl==7.43.0 --global-option="--with-openssl" celery==4.1.1 kombu==4.5.0

【参考方案2】：

对于那些在尝试部署使用 Celery 的 Django 应用程序时尝试解决以下问题的人：

'Hub' 对象没有属性 '_current_http_client' ...

ImportError：curl 客户端需要 pycurl 库。

---

我正在使用以下软件包：

celery==4.4.7 django-celery-beat==2.1.0 django-celery-results==1.2.1

---

解决方案：-

使用 Elastic Beanstalk 的配置文件安装 pycurl，参考下面我的配置文件：

packages:
  yum:
    gcc: []
    gcc-c++: []
    graphviz-devel: []
    libjpeg-turbo-devel: []
    libcurl-devel: []
    openssl-static.x86_64: []

container_commands:
  00_1_pip_upgrade:
    command: /opt/python/run/venv/bin/pip install --upgrade pip
    ignoreErrors: false
  00_2_install_pycurl:
    command: /opt/python/run/venv/bin/pip install pycurl --global-option="--with-openssl" --upgrade
    ignoreErrors: false
    leader_only: true
    .
    .
    .
  xx_add_rest_of_your_commands:

上述代码块中的第二个容器命令应该有助于解决问题。