Django Celery Elastic Beanstalk supervisord 没有这样的进程错误
Posted
技术标签:
【中文标题】Django Celery Elastic Beanstalk supervisord 没有这样的进程错误【英文标题】：Django Celery Elastic Beanstalk supervisord no such process error 【发布时间】：2017-09-14 20:17:42 【问题描述】：.ebextensions 文件脚本中的 celery_config.txt
#!/usr/bin/env bash
# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=$celeryenv%?
# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv
[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A wellfie --loglevel=INFO --workdir=/tmp -S django
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=$celeryenv"
# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf
# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
then
echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi
# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread
# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update
# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd
我的 celery-worker.log
[2017-04-18 18:48:06,042: CRITICAL/MainProcess] Unrecoverable error: ImportError('No module named xxxx.toi3pr.0001.usw2.cache.amazonaws.com',)
Traceback (most recent call last):
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/worker/worker.py", line 203, in start
self.blueprint.start(self)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 115, in start
self.on_start()
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 143, in on_start
self.emit_banner()
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 158, in emit_banner
' \n', self.startup_info(artlines=not use_image))),
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 221, in startup_info
results=self.app.backend.as_uri(),
File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__
value = obj.__dict__[self.__name__] = self.__get(obj)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/base.py", line 1182, in backend
return self._get_backend()
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/base.py", line 900, in _get_backend
self.loader)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/backends.py", line 65, in by_url
return by_name(backend, loader), url
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/backends.py", line 45, in by_name
cls = symbol_by_name(backend, aliases)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named xxxx.toi3pr.0001.usw2.cache.amazonaws.com
我的 celery-beat.log
ImportError: No module named wellfie
Traceback (most recent call last):
File "/opt/python/run/venv/bin/celery", line 9, in <module>
load_entry_point('celery==4.0.2', 'console_scripts', 'celery')()
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/__main__.py", line 14, in main
_main()
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main
cmd.execute_from_commandline(argv)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
super(CeleryCommand, self).execute_from_commandline(argv)))
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 279, in execute_from_commandline
argv = self.setup_app_from_commandline(argv)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
self.app = self.find_app(app)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 503, in find_app
return find_app(app, symbol_by_name=self.symbol_by_name)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/utils.py", line 355, in find_app
sym = symbol_by_name(app, imp=imp)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 506, in symbol_by_name
return imports.symbol_by_name(name, imp=imp)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
module = imp(module_name, package=package, **kwargs)
File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
return imp(module, package=package)
File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
ImportError: No module named wellfie
我的 eb-activity.log
[2017-04-18T19:50:23.354Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/01flip.py] : Starting activity...
[2017-04-18T19:50:28.095Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/01flip.py] : Completed activity. Result:
httpd: stopped
httpd: started
httpd RUNNING pid 11508, uptime 0:00:03
[2017-04-18T19:50:28.095Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Starting activity...
[2017-04-18T19:50:28.195Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Completed activity. Result:
+ chmod 0755 /var/run/httpd
+ /opt/elasticbeanstalk/bin/healthd-track-pidfile --proxy httpd
+ /opt/elasticbeanstalk/bin/healthd-track-pidfile --name application --location /opt/python/run/supervisord.pid
[2017-04-18T19:50:28.195Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2017-04-18T19:50:28.195Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2017-04-18T19:50:28.195Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2017-04-18T19:50:28.487Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",PYTHONPATH="/opt/python/current/app/wellfie:",REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",RDS_DB_NAME="bbbdb",DJANGO_SETTINGS_MODULE="wellfie.settings",RDS_USERNAME="aaaa",RDS_PASSWORD="bbbb",RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"
[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A wellfie --loglevel=INFO --workdir=/tmp -S django
directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10
; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600
; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true
; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998
environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",PYTHONPATH="/opt/python/current/app/wellfie:",REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",RDS_DB_NAME="bbbdb",DJANGO_SETTINGS_MODULE="wellfie.settings",RDS_USERNAME="aaaa",RDS_PASSWORD="bbbb",RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"
No config updates to processes
celeryd: ERROR (no such process)
celeryd: ERROR (no such process)
[2017-04-18T19:50:28.487Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/post.
[2017-04-18T19:50:28.487Z] INFO [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1] : Completed activity. Result:
Application version switch - Command CMD-AppDeploy stage 1 completed
[2017-04-18T19:50:28.487Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter] : Starting activity...
[2017-04-18T19:50:28.487Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation] : Starting activity...
[2017-04-18T19:50:28.488Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation/10-config.sh] : Starting activity...
[2017-04-18T19:50:28.563Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation/10-config.sh] : Completed activity. Result:
Disabled forced hourly log rotation.
[2017-04-18T19:50:28.564Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logpublish/hooks/config.
[2017-04-18T19:50:28.564Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent] : Starting activity...
[2017-04-18T19:50:28.564Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent/10-config.sh] : Starting activity...
[2017-04-18T19:50:28.677Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent/10-config.sh] : Completed activity. Result:
Log streaming option setting is not specified, ignore cloudwatch logs setup.
Disabled log streaming.
[2017-04-18T19:50:28.678Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent] : Completed activity. Result:
Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logstreaming/hooks/config.
[2017-04-18T19:50:28.678Z] INFO [11290] - [Application update app-721b-170418_214950@45/AddonsAfter] : Completed activity.
[2017-04-18T19:50:28.678Z] INFO [11290] - [Application update app-721b-170418_214950@45] : Completed activity. Result:
Application update - Command CMD-AppDeploy succeeded
[2017-04-18T19:50:40.404Z] INFO [11617] - [CMD-TailLogs] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO [11617] - [CMD-TailLogs/AddonsBefore] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO [11617] - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2017-04-18T19:50:40.404Z] INFO [11617] - [CMD-TailLogs/TailLogs] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO [11617] - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity...
我不确定是什么导致了错误，但这条特定的行引起了我的注意 celeryd: ERROR (no such process)。在本地，我的设置就像一个魅力，我得到了所有的任务，运行工人/节拍的问题为零，但在生产中，supervisord 似乎难以访问 celery 工人，或者 celery 工人没有正常启动。
我已经调试了很多，但我在找出问题所在时遇到了问题。我已经删除了变量和网址，它们看起来都还可以。如果我手动 ssh 进入实例，我可以毫无问题地同时运行 celery beat 和 celery worker，但是 supervisord 有问题。
这条路径也正确吗？
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO
我是 AWS 新手，如果此解决方案显而易见，我深表歉意。
编辑：
01_packages.config
packages:
yum:
git: []
postgresql93-devel: []
libcurl-devel: []
02_python.config
container_commands:
04_celery_tasks:
command: "cat .ebextensions/files/celery_configuration.txt > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
05_celery_tasks_run:
command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
leader_only: true
django.config
option_settings:
aws:elasticbeanstalk:application:environment:
DJANGO_SETTINGS_MODULE: "wellfie.settings"
PYTHONPATH: "/opt/python/current/app/wellfie:$PYTHONPATH"
aws:elasticbeanstalk:container:python:
WSGIPath: wellfie/wsgi.py
【问题讨论】：
【参考方案1】：是的，路径没问题：
command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO
我注意到您将 PYTHONPATH 变量设置了两次，就像您会用 eb 环境属性覆盖一样？一旦在应用程序上设置，然后在应用程序 Wellfie 包上设置。
您的日志条目：
environment=
PYTHONPATH="/opt/python/current/app/:",
PATH="/opt/python/run/venv/bin/:%ENV_PATH)s",
RDS_PORT="5432",
PYTHONPATH="/opt/python/current/app/wellfie:",
REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",
RDS_DB_NAME="bbbdb",
DJANGO_SETTINGS_MODULE="wellfie.settings",
RDS_USERNAME="aaaa",
RDS_PASSWORD="bbbb",
RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"
我的：
environment=
PYTHONPATH="/opt/python/current/app/:",
PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",
WEBSERVER="APACHE",
CNAME="xxxxx-prod-env",
PYCURL_SSL_LIBRARY="nss",
APP_ENV="prod"
所以我会从调查开始。 另一件要记住的事情是，如果您使用一些 AWS 服务，例如 Redis，您需要在其上使用该服务的安全组设置打开连接，否则您将无法连接。
注意事项：
如果您在实例上玩游戏，请记住 virtualenv。因此，您可以运行以下命令：
cd /opt/python/current/app && source /opt/python/run/venv/bin/activate && python manage.py
“ImportError: No module named 'wellfie'”可能会发生一次，并被登录到 celery-beat.log, celery-worker.log”，因为系统正在启动并且进程可能还没有准备好。
ImportError('No module named xxxx.toi3pr.0001.usw2.cache.amazonaws.com',) 看起来像 python 初始化问题。
修改文件root_folder/django_app/__init__.py：
from __future__ import absolute_import, unicode_literals
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from django_app.celery import app as celery_app
__all__ = ['celery_app']
修复是改变：
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd
进入
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat
【讨论】：
感谢您的回复。好的，我已经删除了多余的PYTHONPATH。我的安全组在 0.0.0.0/0 和 ://0 的 6379 上有入站规则。关于第二个和第三个 - 你能再解释一下吗？该模块是我环境中的变量。在我的 __init__.py 文件中，我导入了 absolute_import 和 celery_app 。我是否未能在 init.py 文件中包含某些内容？我假设它没有在执行 celery 之前加载我的应用程序？
2. ImportError: No module named 'wellfie'" 也发生在我的节点上，但仅在 eb 重建后发生一次。3。看起来你的工作人员上的 python 执行没有很好地完成 python 初始化。如果你删除了 pythonpath，请尝试在控制台中重建 env .
是的，init 不见了 :) 请稍后查看我的原始帖子
我添加了完全相同的东西，但没有成功 :( 路径是 wellfie/wellfie/__init.py 在本地它可以完美运行。我的 .ebextension 有 01_packages.config、02_python.config、django.config。有可能吗django.config 需要在 02_python.config 之前加载（这包含 celery init 等）我将使用它们三个来编辑原始帖子。我是否将容器命令添加到 django.config 中？
如果您询问 .ebextension，您可以将您的命令添加到那里的任何文件中，只要它具有 .config 扩展名。顺序取决于文件名（如果我没记错的话），然后是命令名，所以最好用数字作为前缀。我真的不知道如何为您提供更多帮助。尝试重建它。还要检查正在部署的内容是否真的是最后提交的版本：在控制台中：“上传和部署”->“应用程序版本页面”。在上一个版本上设置复选框并“部署”。以上是关于Django Celery Elastic Beanstalk supervisord 没有这样的进程错误的主要内容，如果未能解决你的问题，请参考以下文章
由于 ImportError，Celery Django 部署因 Elastic Beanstalk 失败：无法导入名称“Celery”（ElasticBeanstalk::ExternalInvoc
使用 Django 在 Elastic Beanstalk 上运行 Python 的 Celery
使用 Django 在 Elastic Beanstalk 上运行 Python 的 Celery
在 Elastic Beanstalk 上使用 Supervisor 和 Django 将 Celery 作为守护进程运行