AWS Elastic Beanstalk 上的 Celery 配置问题 - “进程没有配置更新”

Posted 2023-02-24

【中文标题】AWS Elastic Beanstalk 上的 Celery 配置问题 - “进程没有配置更新”

【英文标题】：Issues with Celery configuration on AWS Elastic Beanstalk - "No config updates to processes"

【发布时间】：2018-10-07 09:06:55

【问题描述】：

我在 AWS Elastic Beanstalk 上部署了一个 Django 2 应用程序，我正在尝试配置 Celery 以便在同一台机器上执行异步任务。

我的文件：

02_packages.config

files:
  "/usr/local/share/pycurl-7.43.0.tar.gz" :
    mode: "000644"
    owner: root
    group: root
    source: https://pypi.python.org/packages/source/p/pycurl/pycurl-7.43.0.tar.gz

packages:
  yum:
    python34-devel: []
    libcurl-devel: []

commands:
  01_download_pip3:
    # run this before PIP installs requirements as it needs to be compiled with OpenSSL
    command: 'curl -O https://bootstrap.pypa.io/get-pip.py'
  02_install_pip3:
    # run this before PIP installs requirements as it needs to be compiled with OpenSSL
    command: 'python3 get-pip.py'

container_commands:
  03_pycurl_reinstall:
    # run this before PIP installs requirements as it needs to be compiled with OpenSSL
    # the upgrade option is because it will run after PIP installs the requirements.txt file.
    # and it needs to be done with the virtual-env activated
    command: 'source /opt/python/run/venv/bin/activate && pip3 install /usr/local/share/pycurl-7.43.0.tar.gz --global-option="--with-nss" --upgrade'

03_django.config

container_commands:
  01_migrate_db:
    command: "django-admin.py migrate --noinput"
    leader_only: true
  02_createsu: # custom django-admin command to create the "admin" superuser
    command: "source /opt/python/run/venv/bin/activate && python manage.py createsu"
    leader_only: true
  03_update_permissions: # custom django-admin command to update user perms
    command: "source /opt/python/run/venv/bin/activate && python manage.py update_permissions"
    leader_only: true
  04_collectstatic:
    command: "django-admin.py collectstatic --noinput"
  05_pip_upgrade:
    command: /opt/python/run/venv/bin/pip install --upgrade pip
    ignoreErrors: false

option_settings:
  aws:elasticbeanstalk:application:environment:
    DJANGO_SETTINGS_MODULE: "my_proj.settings_prod"
    APP_ENV: "test"
    PYCURL_SSL_LIBRARY: "nss"
  aws:elasticbeanstalk:container:python:
    WSGIPath: myproj/wsgi.py
    NumProcesses: 3
    NumThreads: 20
  aws:elasticbeanstalk:container:python:staticfiles:
    "/static/": "static/"

requirements.txt

boto3==1.6.3
botocore==1.9.3
Django==2.0.3
django-cors-headers==2.2.0
django-filter==1.1.0
django-storages==1.6.5
djangorestframework==3.7.7
djangorestframework-jwt==1.11.0
docutils==0.14
jmespath==0.9.3
Markdown==2.6.11
olefile==0.44
Pillow==5.0.0
psycopg2==2.7.3.2
PyJWT==1.5.3
python-dateutil==2.6.1
pytz==2018.3
reportlab==3.4.0
s3transfer==0.1.13
six==1.11.0
Wand==0.4.4
uwsgi==2.0.17 # WSGI for production deployment
gevent==1.2.2 # Non-blocking Python network library, required by uWSGI
celery==4.1.0
django_celery_beat==1.1.1
django_celery_results==1.0.1

celery_conf/config.py

AWS_ACCESS_KEY_ID = ...
AWS_SECRET_ACCESS_KEY = ...

CELERY_BROKER_TRANSPORT = 'sqs'
CELERY_BROKER_URL = 'sqs://' # 'sqs://%s:%s@' % (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)

CELERY_BROKER_USER = AWS_ACCESS_KEY_ID
CELERY_BROKER_PASSWORD = AWS_SECRET_ACCESS_KEY
CELERY_WORKER_STATE_DB = '/var/run/celery/worker.db'
CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
CELERY_WORKER_PREFETCH_MULTIPLIER = 0 # See https://github.com/celery/celery/issues/3712

CELERY_ACCEPT_CONTENT = ['application/json']
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'

CELERY_DEFAULT_QUEUE = 'myproj-django' # Queue name
CELERY_QUEUES = 
    CELERY_DEFAULT_QUEUE: 
        'exchange': CELERY_DEFAULT_QUEUE,
        'binding_key': CELERY_DEFAULT_QUEUE,
    


CELERY_BROKER_TRANSPORT_OPTIONS = 
    "region": "us-east-1", # US East (N. Virginia)
    'visibility_timeout': 360,
    'polling_interval': 1


CELERY_RESULT_BACKEND = 'django-db'

myproj/celery.py

from __future__ import absolute_import, unicode_literals
import os

from celery import Celery
from celery.schedules import crontab

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproj.settings_prod')

app = Celery('myproj')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

if __name__ == '__main__':
    app.start()

@app.task(bind=True)
def debug_task(self):
        print('Request: 0!r'.format(self.request))

myproj/myapp/tasks.py

from __future__ import absolute_import, unicode_literals
from celery.decorators import task

from celery.utils.log import get_task_logger
logger = get_task_logger(__name__)

@task()
def do_something():
    logger.info('******** CALLING ASYNC TASK WITH CELERY **********')

settings_prod.py

# Importing base settings
from .settings import *

DEBUG = False

# Importing Celery configurations
from celery_conf.config import *
INSTALLED_APPS += ('django_celery_beat',)

更新 1

由于根据 /var/log/celery-beat.log ，看来 celery 无法找到我的项目模块。我认为我的项目结构不是 Celery 所期望的。如何在不改变整个项目结构的情况下使其工作？

我的项目结构如下：

-- myprof-folder/
   -- requirements.txt
   -- .ebextensions/
   -- celery_conf/
      -- __init__.py
      -- config.py
   -- myproj/
      -- __init__.py
      -- settings.py # base settings
      -- settings_prod.py # production settings
      -- urls.py
      -- wsgi.py
      -- myapp1/
         -- models.py
         -- urls.py
         -- apps.py
         -- views.py
         -- tasks.py # here my app's tasks
         -- ...
      -- myapp2/
      -- myapp3/
      -- ...
      -- myappN/

更新 2

99_celery.config 使用 --workdir 选项和 /tmp 作为目录。不需要该选项。我还对该文件进行了一些更改。

99_celery.config

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      # Create required directories
      sudo mkdir -p /var/log/celery/
      sudo mkdir -p /var/run/celery/

      # Create group called 'celery'
      sudo groupadd -f celery
      # add the user 'celery' if it doesn't exist and add it to the group with same name
      id -u celery &>/dev/null || sudo useradd -g celery celery
      # add permissions to the celery user for r+w to the folders just created
      sudo chown -R celery:celery /var/log/celery/
      sudo chown -R celery:celery /var/run/celery/

      # Get django environment variables
      celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
      celeryenv=$celeryenv%?

      # Create celery configuration script
      celeryconf="[program:celeryd-worker]
      ; Set full path to celery program if using virtualenv
      command=/opt/python/run/venv/bin/celery worker -A myproj --loglevel=INFO --logfile="/var/log/celery/%%n%%I.log" --pidfile="/var/run/celery/%%n.pid"

      directory=/opt/python/current/app
      user=celery
      numprocs=1
      stdout_logfile=/var/log/celery-worker.log
      stderr_logfile=/var/log/celery-worker.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 600

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      priority=998

      environment=$celeryenv

      [program:celeryd-beat]
      ; Set full path to celery program if using virtualenv
      command=/opt/python/run/venv/bin/celery beat -A myproj --loglevel=INFO --logfile="/var/log/celery/celery-beat.log" --pidfile="/var/run/celery/celery-beat.pid"

      directory=/opt/python/current/app
      user=celery
      numprocs=1
      stdout_logfile=/var/log/celery-beat.log
      stderr_logfile=/var/log/celery-beat.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 600

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      priority=998

      environment=$celeryenv"

      # Create the celery supervisord conf script
      echo "$celeryconf" | tee /opt/python/etc/celery.conf

      # Add configuration script to supervisord conf (if not there already)
      if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
        then
        echo "[include]" | tee -a /opt/python/etc/supervisord.conf
        echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Enable supervisor to listen for HTTP/XML-RPC requests.
      # supervisorctl will use XML-RPC to communicate with supervisord over port 9001.
      # Source: https://askubuntu.com/questions/911994/supervisorctl-3-3-1-http-localhost9001-refused-connection
      if ! grep -Fxq "[inet_http_server]" /opt/python/etc/supervisord.conf
        then
        echo "[inet_http_server]" | tee -a /opt/python/etc/supervisord.conf
        echo "port = 127.0.0.1:9001" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Reread the supervisord config
      supervisorctl -c /opt/python/etc/supervisord.conf reread

      # Update supervisord in cache without restarting all services
      supervisorctl -c /opt/python/etc/supervisord.conf update

      # Start/Restart celeryd through supervisord
      supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
      supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker


container_commands:
  00_celery_tasks_run:
    command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
    leader_only: true

我的日志：

我通过 SSH 我的 EC2 实例和以下是日志文件：

/var/log/celery-worker.log

Traceback (most recent call last):
  File "/opt/python/run/venv/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/__main__.py", line 14, in main
    _main()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/base.py", line 279, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/base.py", line 503, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/app/utils.py", line 355, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/base.py", line 506, in symbol_by_name
    return imports.symbol_by_name(name, imp=imp)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/opt/python/run/venv/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'myproj'

/var/log/celery-beat.log

Traceback (most recent call last):
  File "/opt/python/run/venv/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/__main__.py", line 14, in main
    _main()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/base.py", line 279, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/base.py", line 503, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/app/utils.py", line 355, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bin/base.py", line 506, in symbol_by_name
    return imports.symbol_by_name(name, imp=imp)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/opt/python/run/venv/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 948, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'myproj'
celery beat v4.1.0 (latentcall) is starting.
__    -    ... __   -        _
LocalTime -> 2018-04-30 19:09:23
Configuration ->
    . broker -> sqs://AKIAJDSLHYFOJ6MYJZ5Q:**@localhost//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> /var/log/celery/celery-beat.log@%INFO
    . maxinterval -> 5.00 seconds (5s)

/var/log/celery/celery.log

[2018-04-30 19:09:24,049: CRITICAL/MainProcess] Unrecoverable error: RuntimeError("Model class django_celery_results.models.TaskResult doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.",)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/bootsteps.py", line 115, in start
    self.on_start()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/apps/worker.py", line 143, in on_start
    self.emit_banner()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/apps/worker.py", line 158, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/apps/worker.py", line 221, in startup_info
    results=self.app.backend.as_uri(),
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/app/base.py", line 1183, in backend
    return self._get_backend()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/app/base.py", line 901, in _get_backend
    self.loader)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/app/backends.py", line 66, in by_url
    return by_name(backend, loader), url
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/celery/app/backends.py", line 46, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/opt/python/run/venv/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django_celery_results/backends/__init__.py", line 4, in <module>
    from .database import DatabaseBackend
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django_celery_results/backends/database.py", line 7, in <module>
    from ..models import TaskResult
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django_celery_results/models.py", line 17, in <module>
    class TaskResult(models.Model):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/models/base.py", line 108, in __new__
    "INSTALLED_APPS." % (module, name)
RuntimeError: Model class django_celery_results.models.TaskResult doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

/var/log/celery/celery-beat.log

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 168, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
    Is the server running on host "127.0.0.1" and accepting
    TCP/IP connections on port 5432?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django_celery_beat/schedulers.py", line 238, in sync
    with transaction.atomic():
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/transaction.py", line 147, in __enter__
    if not connection.get_autocommit():
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 378, in get_autocommit
    self.ensure_connection()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 216, in ensure_connection
    self.connect()
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/backends/base/base.py", line 194, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/opt/python/run/venv/local/lib/python3.6/site-packages/django/db/backends/postgresql/base.py", line 168, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/opt/python/run/venv/local/lib64/python3.6/site-packages/psycopg2/__init__.py", line 130, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
    Is the server running on host "127.0.0.1" and accepting
    TCP/IP connections on port 5432?

【问题讨论】：

你能清理旧的异常并保留最新的异常吗，这有点令人困惑。您如何使用celery_conf？

嗨@TarunLalwani，celery_conf 是一个仅包含config.py 文件的文件夹。它用于settings_prod.py 导入配置：from celery_conf.config import *

查看日志，我得到两个异常：第一个与ModuleNotFoundError 相关，第二个是解析错误。我完全卡住了，我不知道如何解决。

对不起@TarunLalwani 我刚刚清理并更新了代码

celery 错误日志中的错误信息中的模块olem 是什么。 ModuleNotFoundError: No module named 'olem'

【参考方案1】：

所以您的配置存在多个问题

1.芹菜类不正确

command=/opt/python/run/venv/bin/celery worker -A myproj --loglevel=INFO --logfile="/var/log/celery/%%n%%I.log" --pidfile="/var/run/celery/%%n.pid"

按照你的结构应该是这样的

command=/opt/python/run/venv/bin/celery worker -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/%%n%%I.log" --pidfile="/var/run/celery/%%n.pid"

2。 Unicode 破折号而不是 -

您在配置中有一个 unicode en-dash 或其他内容，可能是您从某个网站复制的，而在您的以下命令中它不是 -

command=/opt/python/run/venv/bin/celery worker -A myproj --loglevel=INFO --logfile="/var/log/celery/%%n%%I.log" --pidfile="/var/run/celery/%%n.pid"

3.不附加celerybeat.conf 和celery.conf

下面的代码不工作

if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
 then
    echo "[include]" | tee -a /opt/python/etc/supervisord.conf
    echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi

因为 include 已经是 uswgi.conf 的一部分

4.缺少 inet_server 配置

虽然您的问题如下

if ! grep -Fxq "[inet_http_server]" /opt/python/etc/supervisord.conf
    then
        echo "[inet_http_server]" | tee -a /opt/python/etc/supervisord.conf
        echo "port = 127.0.0.1:9001" | tee -a /opt/python/etc/supervisord.conf
fi

但是您没有使用它，这会导致以下问题

[实例：i-00c786a77c1f5ec11] 实例上的命令失败。返回码：2 输出：（截断）... 错误：已经关闭错误：，：文件：/usr/lib64/python2.7/xmlrpclib.py 行：800 错误：，：文件：/usr/lib64/python2 .7/xmlrpclib.py 行：800。钩子 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh 失败。

最终配置

以下是您需要使用的最终配置

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      # Create required directories
      sudo mkdir -p /var/log/celery/
      sudo mkdir -p /var/run/celery/

      # Create group called 'celery'
      sudo groupadd -f celery
      # add the user 'celery' if it doesn't exist and add it to the group with same name
      id -u celery &>/dev/null || sudo useradd -g celery celery
      # add permissions to the celery user for r+w to the folders just created
      sudo chown -R celery:celery /var/log/celery/
      sudo chown -R celery:celery /var/run/celery/

      # Get django environment variables
      celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/%/%%/g' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
      celeryenv=$celeryenv%?

      # Create CELERY configuration script
      celeryconf="[program:celeryd]
      directory=/opt/python/current/app
      ; Set full path to celery program if using virtualenv
      command=/opt/python/run/venv/bin/celery worker -A celery_conf.celery_app:app --loglevel=INFO --logfile=\"/var/log/celery/%%n%%I.log\" --pidfile=\"/var/run/celery/%%n.pid\"

      user=celery
      numprocs=1
      stdout_logfile=/var/log/celery-worker.log
      stderr_logfile=/var/log/celery-worker.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 60

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      priority=998

      environment=$celeryenv"


      # Create CELERY BEAT configuraiton script
      celerybeatconf="[program:celerybeat]
      ; Set full path to celery program if using virtualenv
      command=/opt/python/run/venv/bin/celery beat -A celery_conf.celery_app:app --loglevel=INFO --logfile=\"/var/log/celery/celery-beat.log\" --pidfile=\"/var/run/celery/celery-beat.pid\"

      directory=/opt/python/current/app
      user=celery
      numprocs=1
      stdout_logfile=/var/log/celerybeat.log
      stderr_logfile=/var/log/celerybeat.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 60

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      priority=999

      environment=$celeryenv"

      # Create the celery supervisord conf script
      echo "$celeryconf" | tee /opt/python/etc/celery.conf
      echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf

      # Add configuration script to supervisord conf (if not there already)
      if ! grep -Fxq "celery.conf" /opt/python/etc/supervisord.conf
        then
          echo "[include]" | tee -a /opt/python/etc/supervisord.conf
          echo "files: uwsgi.conf celery.conf celerybeat.conf" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Enable supervisor to listen for HTTP/XML-RPC requests.
      # supervisorctl will use XML-RPC to communicate with supervisord over port 9001.
      # Source: https://askubuntu.com/questions/911994/supervisorctl-3-3-1-http-localhost9001-refused-connection
      if ! grep -Fxq "[inet_http_server]" /opt/python/etc/supervisord.conf
        then
          echo "[inet_http_server]" | tee -a /opt/python/etc/supervisord.conf
          echo "port = 127.0.0.1:9001" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Reread the supervisord config
      supervisorctl -c /opt/python/etc/supervisord.conf reread

      # Update supervisord in cache without restarting all services
      supervisorctl -c /opt/python/etc/supervisord.conf update

      # Start/Restart celeryd through supervisord
      supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd
      supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat


commands:
  01_killotherbeats:
    command: "ps auxww | grep 'celery beat' | awk 'print $2' | sudo xargs kill -9 || true"
    ignoreErrors: true
  02_restartbeat:
    command: "supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat"
    leader_only: true

案件已解决！！

【讨论】：

配置文件现在可以正常工作，我终于可以在 Elastic Beanstalk 上将 Celery 与 Django 一起使用。但是，我问了一个新问题，因为我仍然收到error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>（在这个问题上没有提到），但只有在重新部署环境时才可以

如果你能帮到我，请查看***.com/questions/50135628/…

你发布了同一个问题的链接

感谢您的回答。它也几乎解决了我的问题。但是当我使用你的解决方案时，当我开始 celery beat 时出现权限被拒绝错误。 celery 用户没有所有必要的权限，这就是发生 celery beat 问题的原因。在这里我描述了我是如何解决这个问题的***.com/a/51194596/2983211