芹菜节拍不接周期性任务
Posted
技术标签:
【中文标题】芹菜节拍不接周期性任务【英文标题】:Celery beat not picking up periodic tasks 【发布时间】:2018-04-05 10:05:07 【问题描述】:我正在尝试开始使用 celery,但我无法启动并运行我的任务。我已经安装了django-celery-beat
和celery4
。
我的设置文件。
已安装的应用程序(带有 celery 包)
...
'django_celery_beat',
'django_celery_results'
芹菜配置
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
芹菜.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'sandbox.settings')
app = Celery('sandbox')
# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
# should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')
# Load task modules from all registered Django app configs.
app.autodiscover_tasks()
@app.task(bind=True)
def debug_task(self):
print('Request: 0!r'.format(self.request))
还有我的简单任务,我将其配置为通过 django celery beat 的管理面板运行。
from __future__ import absolute_import, unicode_literals
from sandbox.celery import app
@app.task()
def try_celery():
print "Trying out Celery"
我正在尝试将此任务作为定期任务(节拍)运行,其中 cron 选项卡为 */2 * * * *
我得到的日志是,
$ celery -A sandbox worker --loglevel=debug
[2017-10-24 14:28:02,999: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-10-24 14:28:03,001: DEBUG/MainProcess] | Worker: Building graph...
[2017-10-24 14:28:03,002: DEBUG/MainProcess] | Worker: New boot order: Beat, Timer, Hub, Pool, Autoscaler, StateDB, Consumer
[2017-10-24 14:28:03,017: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-10-24 14:28:03,017: DEBUG/MainProcess] | Consumer: Building graph...
[2017-10-24 14:28:03,038: DEBUG/MainProcess] | Consumer: New boot order: Connection, Events, Mingle, Tasks, Control, Agent, Gossip, Heart, event loop
-------------- celery@mypc v4.1.0 (latentcall)
---- **** -----
--- * *** * -- Linux-4.9.0-kali3-amd64-x86_64-with-Kali-kali-rolling-kali-rolling 2017-10-24 14:28:03
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: sandbox:0x7fe519d38610
- ** ---------- .> transport: redis://localhost:6379/0
- ** ---------- .> results: redis://localhost:6379/
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. celery.accumulate
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
. sandbox.applications.cats.try_celery
. sandbox.celery.debug_task
[2017-10-24 14:28:03,053: DEBUG/MainProcess] | Worker: Starting Hub
[2017-10-24 14:28:03,053: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:03,053: DEBUG/MainProcess] | Worker: Starting Pool
[2017-10-24 14:28:03,197: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:03,198: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-10-24 14:28:03,199: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-10-24 14:28:03,216: INFO/MainProcess] Connected to redis://localhost:6379/0
[2017-10-24 14:28:03,217: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:03,217: DEBUG/MainProcess] | Consumer: Starting Events
[2017-10-24 14:28:03,228: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:03,229: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-10-24 14:28:03,229: INFO/MainProcess] mingle: searching for neighbors
[2017-10-24 14:28:04,255: INFO/MainProcess] mingle: all alone
[2017-10-24 14:28:04,256: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,256: DEBUG/MainProcess] | Consumer: Starting Tasks
[2017-10-24 14:28:04,273: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,274: DEBUG/MainProcess] | Consumer: Starting Control
[2017-10-24 14:28:04,277: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,277: DEBUG/MainProcess] | Consumer: Starting Gossip
[2017-10-24 14:28:04,281: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,282: DEBUG/MainProcess] | Consumer: Starting Heart
[2017-10-24 14:28:04,284: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,284: DEBUG/MainProcess] | Consumer: Starting event loop
[2017-10-24 14:28:04,285: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2017-10-24 14:28:04,294: WARNING/MainProcess] /home/alexd/.virtualenvs/skate/local/lib/python2.7/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2017-10-24 14:28:04,294: INFO/MainProcess] celery@mypc ready.
[2017-10-24 14:28:04,295: DEBUG/MainProcess] basic.qos: prefetch_count->8
为什么我的任务不起作用?
更新
以节拍运行 celery 的日志
$ celery -A sandbox beat --loglevel=debug
celery beat v4.1.0 (latentcall) is starting.
Stale pidfile exists - Removing it.
__ - ... __ - _
LocalTime -> 2017-10-24 15:07:20
Configuration ->
. broker -> redis://localhost:6379/0
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]@%DEBUG
. maxinterval -> 5.00 minutes (300s)
[2017-10-24 15:07:20,216: DEBUG/MainProcess] Setting default socket timeout to 30
[2017-10-24 15:07:20,217: INFO/MainProcess] beat: Starting...
[2017-10-24 15:07:20,372: DEBUG/MainProcess] Current schedule:
[2017-10-24 15:07:20,373: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2017-10-24 15:07:20,373: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
[2017-10-24 15:15:43,232: DEBUG/MainProcess] beat: Synchronizing schedule...
[2017-10-24 15:15:43,245: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
创建任务 这就是在管理员中将任务创建为周期性任务的方式。
ps。我更改为间隔选项,我的 cron 选项卡可能有问题吗?反正还是不行。
【问题讨论】:
根据documentation,要启动芹菜节拍时间表,您需要执行$celery -A proj beat
-- 看起来问题可能是您指定了worker
而不是beat
。
我看到您使用节拍调度程序的输出更新了您的问题。看起来您正在使用默认的celery.beat.PersistentScheduler
。 According to the docs,对于这个扩展,你想使用数据库调度器。因此,也许尝试添加选项以指定数据库调度程序,因为文档建议 --scheduler django_celery_beat.schedulers:DatabaseScheduler
。看看这是否有所作为。另外,不要忘记执行数据库迁移。
谢谢@sytech,这解决了我的问题。你会写一个答案,以便我可以接受并关闭这个问题吗?
是否可以将其添加到设置文件中?
是的,文档确实在页面底部提到了Note: You may also add this as an settings option directly.
快速查看docs,似乎指定beat_scheduler="django_celery_beat.schedulers:DatabaseScheduler"
应该可以解决问题。
【参考方案1】:
在“沙盒”项目中的“init.py”中添加以下代码,以便为 Django 管理员挑选任务。
这将确保在 django 启动时始终导入应用程序,以便该任务将使用此应用程序。
from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app',)
【讨论】:
【参考方案2】:在这种情况下,根本原因是节拍调度程序需要使用适当的参数启动。您提供了以下命令:
$ celery -A sandbox worker --loglevel=debug
但是,要以 beat 时间表启动 celery,(与常规 celery worker 不同)您必须指定 beat
而不是 worker
。而且,在使用django_celery_beat
扩展时,需要使用数据库调度器django_celery_beat.schedulers:DatabaseScheduler
而不是默认调度器celery.beat.PersistentScheduler
。
所以更正的命令是:
$ celery -A sandbox beat --loglevel=debug --scheduler django_celery_beat.schedulers:DatabaseScheduler
Supporting documentation
【讨论】:
从文档看,--scheduler
后面应该没有=
的符号。 docs.celeryproject.org/en/latest/userguide/…。不太确定这是否会影响。
据我所知,解析器对此并不挑剔。但由于这就是文档中反映的内容,我认为它值得改变。如果您想编辑我的答案,我很乐意接受 @ailionx 的更改——否则我最终会解决它。【参考方案3】:
我认为你没有定义 cron shcedule。它存储在哪里?通常它在磁盘或数据库中(django_celery)。见http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html
此外,您必须使用节拍选项运行您的工作人员
【讨论】:
这一定是评论,我已经更新了问题。以上是关于芹菜节拍不接周期性任务的主要内容,如果未能解决你的问题,请参考以下文章
Celery---一个懂得异步任务,延时任务,周期任务的芹菜
Celery - 一个懂得 异步任务 , 定时任务 , 周期任务 的芹菜