在 Elastic Beanstalk(AWS) 中守护 Celerybeat

Posted 2023-02-24

【中文标题】在 Elastic Beanstalk(AWS) 中守护 Celerybeat

【英文标题】：Daemonize Celerybeat in Elastic Beanstalk(AWS)

【发布时间】：2015-05-26 04:13:00

【问题描述】：

我正在尝试将 celerybeat 作为 Elastic beanstalk 中的守护进程运行。这是我的配置文件：

files:
"/opt/python/log/django.log":
mode: "000666"
owner: ec2-user
group: ec2-user
content: |
  # Log file
encoding: plain
"/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
mode: "000755"
owner: root
group: root
content: |
  #!/usr/bin/env bash
  # Get django environment variables
  celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/%/%%/g' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
  celeryenv=$celeryenv%?

  # Create celery configuraiton script
  celeryconf="[program:celeryd]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A avtotest --loglevel=INFO

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=$celeryenv"

  # Create celerybeat configuraiton script
  celerybeatconf="[program:celerybeat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A avtotest --loglevel=INFO

  ; remove the -A avtotest argument if you are not using an app instance

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celerybeat.log
  stderr_logfile=/var/log/celerybeat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=999

  environment=$celeryenv"

  # Create the celery and beat supervisord conf script
  echo "$celeryconf" | tee /opt/python/etc/celery.conf
  echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf

  # Add configuration script to supervisord conf (if not there already)
  if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
      then
      echo "[include]" | tee -a /opt/python/etc/supervisord.conf
      echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
      echo "files: celerybeat.conf" | tee -a /opt/python/etc/supervisord.conf
  fi

  # Reread the supervisord config
  supervisorctl -c /opt/python/etc/supervisord.conf reread

  # Update supervisord in cache without restarting all services
  supervisorctl -c /opt/python/etc/supervisord.conf update

  # Start/Restart celeryd through supervisord
  supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd

这个文件守护着 celery 和 celerybeat。芹菜工作正常。但 celerybeat 不是。我没有看到创建的 celerybeat.log 文件，我认为这表明 celerybeat 无法正常工作。

对此有什么想法吗？

如果需要，我会发布更多代码。感谢您的帮助

【问题讨论】：

您是否也应该在脚本的最后一行重新启动celerybeat？

【参考方案1】：

你的 supervisord 语法有点不对，首先你可能需要 SSH 到你的实例，并直接编辑 supervisord.conf 文件（vim /opt/python/etc/supervisord.conf），并直接修复这一行.

echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
echo "files: celerybeat.conf" | tee -a /opt/python/etc/supervisord.conf

应该是

echo "[include]" | tee -a /opt/python/etc/supervisord.conf
echo "files: celery.conf celerybeat.conf" | tee -a /opt/python/etc/supervisord.conf

编辑：

要运行 celerybeat，并确保它只在你的所有机器上运行一次，你应该将这些行放在你的配置文件中 --

04_killotherbeats:
  command: "ps auxww | grep 'celery beat' | awk 'print $2' | sudo xargs kill -9 || true"
05_restartbeat:
  command: "supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat"
  leader_only: true

【讨论】：

我还必须 (1) 将 --pidfile=/tmp/celerybeat.pid 添加到 celerybeat.conf 中的 Celerybeat 命令和 (2) 将 ignoreErrors: true 添加到 04_killotherbeats 以使其在 Elastic Beanstalk 工作实例上正常工作.

@HakanB。你能详细说明一下，因为我的 celerybeat 恶魔被随机杀死并且拒绝开始引用未找到的 supervisord 吗？

一切正常，但 celerybeat 在每个实例上都提供午餐。结果就是周期性任务的重复……

@MatthieuDELMAIRE 如果你有 leader_only: true，那么这不应该发生。这个设置的问题是，如果你使用自动缩放实例，你的领导者实例可能会被杀死。

@jonzlin95 我知道，这就是为什么它很奇怪，当我 ssh 进入实例时，我可以在每个实例上看到 celery beat 过程