定时任务调度-Celery

Posted liujiliang

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了定时任务调度-Celery相关的知识,希望对你有一定的参考价值。

确保任务不重叠解决方法:

from celery import task
from celery.five import monotonic
from celery.utils.log import get_task_logger
from contextlib import contextmanager
from django.core.cache import cache
from hashlib import md5
from djangofeeds.models import Feed

logger = get_task_logger(__name__)

LOCK_EXPIRE = 60 * 10  # Lock expires in 10 minutes

@contextmanager
def memcache_lock(lock_id, oid):
    timeout_at = monotonic() + LOCK_EXPIRE - 3
    # cache.add fails if the key already exists
    status = cache.add(lock_id, oid, LOCK_EXPIRE)
    try:
        yield status
    finally:
        # memcache delete is very slow, but we have to use it to take
        # advantage of using add() for atomic locking
        if monotonic() < timeout_at and status:
            # don‘t release the lock if we exceeded the timeout
            # to lessen the chance of releasing an expired lock
            # owned by someone else
            # also don‘t release the lock if we didn‘t acquire it
            cache.delete(lock_id)

@task(bind=True)
def import_feed(self, feed_url):
    # The cache key consists of the task name and the MD5 digest
    # of the feed URL.
    feed_url_hexdigest = md5(feed_url).hexdigest()
    lock_id = ‘{0}-lock-{1}‘.format(self.name, feed_url_hexdigest)
    logger.debug(‘Importing feed: %s‘, feed_url)
    with memcache_lock(lock_id, self.app.oid) as acquired:
        if acquired:
            return Feed.objects.import_feed(feed_url).url
    logger.debug(
        ‘Feed %s is already being imported by another worker‘, feed_url)

  

以上是关于定时任务调度-Celery的主要内容,如果未能解决你的问题,请参考以下文章

Celery完成定时任务

CELERY 定时任务

python中的轻量级定时任务调度库:schedule

celery执行异步任务和定时任务

python中的轻量级定时任务调度库:schedule

44. Python Celery多实例 定时任务