在 Elastic Beanstalk 上使用 Supervisor 和 Django 将 Celery 作为守护进程运行

Posted 2023-03-04

技术标签:

【中文标题】在 Elastic Beanstalk 上使用 Supervisor 和 Django 将 Celery 作为守护进程运行【英文标题】：Running Celery as Daemon with Supervisor and Django on Elastic Beanstalk 【发布时间】：2017-10-31 07:12:37 【问题描述】：

我正在尝试让 Celery 使用 Django 在 EB 环境中运行，并且我已经非常接近让它全部运行。我有这个主管的配置文件

#!/usr/bin/env bash

# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=$celeryenv%?

# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO

directory=/opt/python/current/app
user=ec2-user
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv,USER="ec2-user"

[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv"

# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf

# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi

# Reread the supervisord config
echo reread
supervisorctl -c /opt/python/etc/supervisord.conf reread

echo reload
# supervisorctl -c /opt/python/etc/supervisord.conf reload

# Update supervisord in cache without restarting all services
echo update
supervisorctl -c /opt/python/etc/supervisord.conf update

# Start/Restart celeryd through supervisord
echo restart
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker

就目前而言，当我部署它时，celery beat 将运行（通过 sshing 验证并使用 ps -aux | less 查看正在运行的进程）但没有 celeryd-worker。在我的 activity.log 中有这个。

[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Completed activity. Result:
  + chmod 0755 /var/run/httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --proxy httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --name application --location /opt/python/run/supervisord.pid
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2017-05-30T17:23:15.243Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2017-05-30T17:23:37.675Z] INFO  [8806]  - [Application update app-cebf-170530_101825@6/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
  [program:celeryd-worker]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A unite --loglevel=INFO

  directory=/opt/python/current/app
  user=ec2-user
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"

  [program:celeryd-beat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A unite --loglevel=INFO --workdir=/tmp -S django

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-beat.log
  stderr_logfile=/var/log/celery-beat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",DJANGO_SETTINGS_MODULE="settings"
  reread
  No config updates to processes
  reload
  update
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: ERROR (not running)
  celeryd-worker: ERROR (abnormal termination)

所以当我尝试启动 worker 但不是 beat 时出现问题。

如果我 SSH 进入 EB 并查看 /var/log/celery-worker.log 文件，我会看到这个。

[2017-05-30 12:30:21,370: CRITICAL/MainProcess] Unrecoverable error: ImportError('The curl client requires the pycurl library.',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 20, in get_client
    return hub._current_http_client
AttributeError: 'Hub' object has no attribute '_current_http_client'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 584, in start
    c.loop(*c.loop_args())
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/celery/worker/loops.py", line 88, in asynloop
    next(loop)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/hub.py", line 282, in create_loop
    item()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 139, in __call__
    return self.throw()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/vine/promises.py", line 136, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 290, in _schedule_queue
    queue, callback=promise(self._loop1, (queue,)),
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 306, in _get_bulk_async
    return self._get_async(queue, maxcount, callback=callback)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 315, in _get_async
    q, count=count, connection=self.asynsqs,
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 412, in asynsqs
    AsyncSQSConnection, _asynsqs.regions(),
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/transport/SQS.py", line 400, in _aws_connect_to
    port=port)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/sqs/connection.py", line 39, in __init__
    https_connection_factory=https_connection_factory, **kwargs
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 223, in __init__
    AsyncConnection.__init__(self, http_client, **http_client_params)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 172, in __init__
    self._httpclient = http_client or get_client()
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 22, in get_client
    client = hub._current_http_client = Client(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/__init__.py", line 13, in Client
    return CurlClient(hub, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.4/site-packages/kombu/async/http/curl.py", line 43, in __init__
    raise ImportError('The curl client requires the pycurl library.')
ImportError: The curl client requires the pycurl library.

我在本地计算机上执行此操作时也遇到了这种情况。虽然我已经确定 curl 是使用 nss 和这个容器命令安装的。

container_commands:
  01_libcurl_ssl_backend:
    command: "export PYCURL_SSL_LIBRARY=nss"
    leader_only: true

这是 pycurl 正在寻找的。我的需求文件中也有 pycurl，如果我 pip freeze 我可以看到 pycurl 已安装。我通过重新创建我的虚拟环境在本地解决了这个问题。但不太确定在这里做什么。

编辑 1 我在

中添加

PYCURL_SSL_LIBRARY: nss

到我的配置文件中的选项设置，并且能够部署并显示略有不同的行为。之后查看活动日志时。

代替

update
restart
celeryd-beat: stopped
celeryd-beat: started
celeryd-worker: ERROR (not running)
celeryd-worker: ERROR (abnormal termination)

我明白了：

    reread
  celeryd-beat: changed
  celeryd-worker: changed
  reload
  update
  celeryd-beat: stopped
  celeryd-beat: updated process group
  celeryd-worker: stopped
  celeryd-worker: updated process group
  restart
  celeryd-beat: stopped
  celeryd-beat: started
  celeryd-worker: stopped
  celeryd-worker: ERROR (abnormal termination)

但是 celery-worker.log 中仍然有关于 pycurl 的相同错误

【问题讨论】：

要强制重建虚拟环境，您可以转到您的 EC2 实例，然后终止由 Elasticbeanstalk 管理的实例。实例将重建（需要一段时间）。 @jonzlin95 你说得对，这确实需要一段时间。但是在意识到我可以将环境变量设置在其他可能是问题的地方之后，我将尝试一下。也尝试从 Elasticbeanstalk 控制台设置环境变量。谢谢！我现在就试试。只需部署具有更新代码的新版本，然后将重建.. @jonzlin95 所以这不起作用。 【参考方案1】：

尝试 SSH 到您的实例并自己导入 pycurl，使用 EB 用于运行您的应用程序的同一 python。然后你会看到实际的错误。

例如，这是我在没有--global-option="--with-nss" 的情况下构建 pycurl 时得到的：

[root@ip-172-31-2-149 log]# /opt/python/run/venv/local/bin/python Python 3.4.3 (default, Sep 1 2016, 23:33:38) [GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux Type "help", "copyright", "credits" or "license" for more information. \>>> import pycurl Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: pycurl: libcurl link-time ssl backend (nss) is different from compile-time ssl backend (none/other)

【讨论】：

使用 pip 安装时使用export PYCURL_SSL_LIBRARY=nss && pip install --compile pycurl（将nss 替换为正确的库）。详细on the documentation.【参考方案2】：

1 - 关注这个问题的答案：https://***.com/a/41097851/3407986

2 - 您必须在包 eb 配置文件/部分中添加 libcurl-devel，请参阅我的 .ebextensions/02_packages.config 文件：

packages:
  yum:
    // ... your another packages where ...
    // add libcurl-devel, dependency of pycurl package
    libcurl-devel: []

【讨论】：

以上是关于在 Elastic Beanstalk 上使用 Supervisor 和 Django 将 Celery 作为守护进程运行的主要内容，如果未能解决你的问题，请参考以下文章

在 Elastic Beanstalk 上使用 Yarn 安装包

AWS S3 行为 - 本地主机 v/s Elastic Beanstalk？

使用 cloudformation 在 Elastic Beanstalk 上启动 docker 多容器

使用 Postgres 在 Elastic Beanstalk 上设置 Django

使用 Docker 在 AWS Elastic Beanstalk 上部署失败

在 Elastic Beanstalk 上使用 Nginx 在子域上配置 Laravel