压测：celey backend为rabbitmq pk redis

Posted 2020-09-26 jihite

使用celery的backend异步获取结果，本文使用rabbitmq 和 redis分别作为backend，代码对比如下

from celery import Celery, platforms
import time
import os
from datetime import datetime

app = Celery(‘proj‘,
             broker=‘amqp://admin:[email protected]:5672‘,     //rabbitmq
             backend=‘amqp://admin:[email protected]:5672‘,    //redis
             //backend=‘redis://ip:10013/0‘,
             include=[‘tasks‘]
             )
app.conf.update(
    CELERYD_PREFETCH_MULTIPLIER=1,
    CELERY_ACKS_LAT = True,
)

@app.task
def fun_1(n):
    return {"task_id": "0a14c9ac91de419880c4499a8f57418e",
            "status": 0,
            "desc": str(n) }

@app.task
def chk_total_callback(tasks_rets):
    end_time = datetime.now()
    print "end_time", end_time
    return 0


if __name__ == "__main__":
    app.start()

发送消息

from tasks import *                                                    
from celery import group, chain                                        
from datetime import datetime                                          
import time                                                            
                                                                       
                                                                       
#job_group = group([fun_1.s(i) for i in range(30000)])                 
beg_time = datetime.now()                                              
for j in range(8000):
    job_group = group([fun_1.s(i) for i in range(20)])                 
    chain(job_group, chk_total_callback.s())()                         
    #time.sleep(0.25)                                                  
                                                                       
print "beg_time:", beg_time

压测场景一：

注：1000*20：一次发送1000次，分片20

力度	backend	耗时	内存峰值	均值
1000*20	rabbitmq	18:40:36-18:41:21	0.504G	45ms
	redis	18:48:37-18:51:21	11.84M	82ms
2000*20	rabbitmq	21:57:06 -21:58:41	1.008G	47.5ms
	redis	18:58:52-19:04:10	20.26M	159ms
3000*20	rabbitmq	21:51:40-21:54:03	1.512G	47.6ms
	redis	19:43:31-19:51:41	28.18M	163ms
4000*20	rabbitmq	20:59:12-21:02:28	2.268G	49ms
	redis	20:00:00-20:10:52	37.10M	163ms

压测场景二：

500*20: 500次，每次间隔1s，分片20

力度	backend	耗时	内存峰值	耗时
500*20	rabbitmq	22:34:28-22:43:01	0.504G	513s
	redis	22:49:23-22:57:56	7.63M	513s

压测场景三：

8000 * 0.25 = 2000s, 任务数：8000   8000 * 20 + 8000 = 8000 * 21=168000 * 2=336000

力度	backend	耗时	内存峰值	耗时
8000 * 0.25 = 2000s *20	rabbitmq	10:51:38-11:28:06	1.26G	2188s
	redis	11:56:17-12:33:00	65.96M	2203s
16000 * 0.25 = 4000s *20	rabbitmq	18:26:39-19:39:37	4.28G	73*60=4380
	redis	20:13:57-22:49:33	130.69M	9360s