多个celery如何使用同一个redis做为broker？

Posted 2021-12-30 shengulong

• 曾经有个哥们提了一个这方面的requests，但是最终没有合入的celery中去，所以目前celery没有这个功能，祥见：
  1. https://github.com/celery/kombu/pull/912
  2. https://github.com/celery/kombu/issues/853
     主要思路是给key加前缀。公司java语言的redis client已经实现了加前缀的功能，这个各个应用就不会出现使用相同key的问题

• 这个方案可以参考Run multiple Celeries on a single Redis

• 另外可以通过使用不同的redis的db号来区分，各个应用使用各自的db来实现隔离

Run multiple Celeries on a single Redis

Celery is a great tool for running asynchronous Django tasks, but it can be complicated to configure. One use case I often face is running multiple web applications on the same server, each with their own Celery daemon.

The web apps are typically completely unrelated and may be running different versions of Django, Celery and other packages in separate virtualenvs. For this reason, I also want to keep their Celery backends separated.

There are a quite a few ways to do it:

+ Use a database server (SQL or NoSQL) as Celery's backend, and use separate databases for each app. This is often the default solution, but not the most effective one.
+ Use a message queue server as Celery's backend, and use separate queues for each app. You will need to choose a queue server and install it just for this purpose.
+ Use Redis as Celery's backend, and use separate Redis database numbers for each app. You will need to coordinate so that each app has a unique database number, which can be quite tiresome.
+ Use Redis, and use the same database number but separate queue names for each app. Easy, effective and simple! This is also nice for local development, since you usually already have a Redis server running on your laptop and it needs no configuration.

Using Redis with separate queue names

To use a local Redis server as the Celery backend, all you need in Django‘s settings.py is this:

BROKER_URL = 'redis://'
CELERY_DEFAULT_QUEUE = 'myapp'

Another web application would then use a different name for the queue:

BROKER_URL = 'redis://'
CELERY_DEFAULT_QUEUE = 'anotherapp'

When you run the Celery daemon processes, each just needs to know which queue it‘s watching:

manage.py celeryd -Q myapp
manage.py celeryd -Q anotherapp

How is all this stored in the Redis database? You will see a new entry appear:

1) "_kombu.binding.celery"

Under that key, which is a SET, you can see the names of all the configured queues as something like this (use the Redis SMEMBERS command):

1) "celery\x06\x16\x06\x16myapp"
2) "celery\x06\x16\x06\x16anotherapp"

Whenever a new task is created, a Redis key temporarily appears for its queue:

2) "myapp"

As the Celery daemon picks it up, the key is immediately deleted so you won‘t usually see it unless you stop the daemon.

Note that the queue names are used as a top level Redis keys without any prefixes, so you should choose them wisely.