Python并发编程-RabbitMQ队列

Posted 2020-08-29

RabbitMQ队列　　

安装 http://www.rabbitmq.com/install-standalone-mac.html

安装python rabbitMQ module 

1 2 3 4 5 6 7

pip install pika or easy_install pika or 源码    https://pypi.python.org/pypi/pika

实现最简单的队列通信

send端

 1 #!/usr/bin/env python
 2 import pika
 3  
 4 connection = pika.BlockingConnection(pika.ConnectionParameters(
 5                ‘localhost‘))
 6 channel = connection.channel()
 7  
 8 #声明queue
 9 channel.queue_declare(queue=‘hello‘)
10  
11 #n RabbitMQ a message can never be sent directly to the queue, it always needs to go through an exchange.
12 channel.basic_publish(exchange=‘‘,
13                       routing_key=‘hello‘,
14                       body=‘Hello World!‘)
15 print(" [x] Sent ‘Hello World!‘")
16 connection.close()

receive端

 1 #!/usr/bin/env python
 2 # -*- coding:utf-8 -*- 
 3 #Author: nulige
 4 
 5 import pika
 6 
 7 connection = pika.BlockingConnection(pika.ConnectionParameters(
 8     ‘localhost‘))
 9 channel = connection.channel()
10 
11 # You may ask why we declare the queue again ? we have already declared it in our previous code.
12 # We could avoid that if we were sure that the queue already exists. For example if send.py program
13 # was run before. But we‘re not yet sure which program to run first. In such cases it‘s a good
14 # practice to repeat declaring the queue in both programs.
15 channel.queue_declare(queue=‘hello‘)
16 
17 
18 def callback(ch, method, properties, body):
19     print(" [x] Received %r" % body)
20 
21 
22 channel.basic_consume(callback,
23                       queue=‘hello‘,
24                       no_ack=True)
25 
26 print(‘ [*] Waiting for messages. To exit press CTRL+C‘)
27 channel.start_consuming()

Work Queues

在这种模式下，RabbitMQ会默认把p发的消息依次分发给各个消费者(c),跟负载均衡差不多。

消息提供者代码

 1 import pika
 2 import time
 3 connection = pika.BlockingConnection(pika.ConnectionParameters(
 4     ‘localhost‘))
 5 channel = connection.channel()
 6  
 7 # 声明queue
 8 channel.queue_declare(queue=‘task_queue‘)
 9  
10 # n RabbitMQ a message can never be sent directly to the queue, it always needs to go through an exchange.
11 import sys
12  
13 message = ‘ ‘.join(sys.argv[1:]) or "Hello World! %s" % time.time()
14 channel.basic_publish(exchange=‘‘,
15                       routing_key=‘task_queue‘,
16                       body=message,
17                       properties=pika.BasicProperties(
18                           delivery_mode=2,  # make message persistent
19                       )
20                       )
21 print(" [x] Sent %r" % message)
22 connection.close()

消费者代码

 1 #_*_coding:utf-8_*_
 2  
 3 import pika, time
 4  
 5 connection = pika.BlockingConnection(pika.ConnectionParameters(
 6     ‘localhost‘))
 7 channel = connection.channel()
 8  
 9  
10 def callback(ch, method, properties, body):
11     print(" [x] Received %r" % body)
12     time.sleep(20)
13     print(" [x] Done")
14     print("method.delivery_tag",method.delivery_tag)
15     ch.basic_ack(delivery_tag=method.delivery_tag)
16  
17  
18 channel.basic_consume(callback,
19                       queue=‘task_queue‘,
20                       no_ack=True
21                       )
22  
23 print(‘ [*] Waiting for messages. To exit press CTRL+C‘)
24 channel.start_consuming()

此时，先启动消息生产者，然后再分别启动3个消费者，通过生产者多发送几条消息，你会发现，这几条消息会被依次分配到各个消费者身上　　

Doing a task can take a few seconds. You may wonder what happens if one of the consumers starts a long task and dies with it only partly done. With our current code once RabbitMQ delivers message to the customer it immediately removes it from memory. In this case, if you kill a worker we will lose the message it was just processing. We‘ll also lose all the messages that were dispatched to this particular worker but were not yet handled.

But we don‘t want to lose any tasks. If a worker dies, we‘d like the task to be delivered to another worker.

In order to make sure a message is never lost, RabbitMQ supports message acknowledgments. An ack(nowledgement) is sent back from the consumer to tell RabbitMQ that a particular message had been received, processed and that RabbitMQ is free to delete it.

If a consumer dies (its channel is closed, connection is closed, or TCP connection is lost) without sending an ack, RabbitMQ will understand that a message wasn‘t processed fully and will re-queue it. If there are other consumers online at the same time, it will then quickly redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally die.

There aren‘t any message timeouts; RabbitMQ will redeliver the message when the consumer dies. It‘s fine even if processing a message takes a very, very long time.

Message acknowledgments are turned on by default. In previous examples we explicitly turned them off via the no_ack=True flag. It‘s time to remove this flag and send a proper acknowledgment from the worker, once we‘re done with a task.

1 def callback(ch, method, properties, body):
2     print " [x] Received %r" % (body,)
3     time.sleep( body.count(‘.‘) )
4     print " [x] Done"
5     ch.basic_ack(delivery_tag = method.delivery_tag)
6  
7 channel.basic_consume(callback,
8                       queue=‘hello‘)

Using this code we can be sure that even if you kill a worker using CTRL+C while it was processing a message, nothing will be lost. Soon after the worker dies all unacknowledged messages will be redelivered。

消息持久化　　

We have learned how to make sure that even if the consumer dies, the task isn‘t lost(by default, if wanna disable  use no_ack=True). But our tasks will still be lost if RabbitMQ server stops.

When RabbitMQ quits or crashes it will forget the queues and messages unless you tell it not to. Two things are required to make sure that messages aren‘t lost: we need to mark both the queue and messages as durable.

First, we need to make sure that RabbitMQ will never lose our queue. In order to do so, we need to declare it as durable:

1	channel.queue_declare(queue=‘hello‘, durable=True)

　　

Although this command is correct by itself, it won‘t work in our setup. That‘s because we‘ve already defined a queue called hello which is not durable. RabbitMQ doesn‘t allow you to redefine an existing queue with different parameters and will return an error to any program that tries to do that. But there is a quick workaround - let‘s declare a queue with different name, for exampletask_queue:

1	channel.queue_declare(queue=‘task_queue‘, durable=True)

　　

This queue_declare change needs to be applied to both the producer and consumer code.

At that point we‘re sure that the task_queue queue won‘t be lost even if RabbitMQ restarts. Now we need to mark our messages as persistent - by supplying a delivery_mode property with a value 2.

1 2 3 4 5 6

channel.basic_publish(exchange=‘‘,                       routing_key="task_queue",                       body=message,                       properties=pika.BasicProperties(                          delivery_mode = 2, # make message persistent                       ))

消息公平分发

如果Rabbit只管按顺序把消息发到各个消费者身上，不考虑消费者负载的话，很可能出现，一个机器配置不高的消费者那里堆积了很多消息处理不完，同时配置高的消费者却一直很轻松。为解决此问题，可以在各个消费者端，配置perfetch=1,意思就是告诉RabbitMQ在我这个消费者当前消息还没处理完的时候就不要再给我发新消息了。

1 channel.basic_qos(prefetch_count=1)

生产者端带消息持久化+公平分发的完整代码

 1 #!/usr/bin/env python
 2 import pika
 3 import sys
 4  
 5 connection = pika.BlockingConnection(pika.ConnectionParameters(
 6         host=‘localhost‘))
 7 channel = connection.channel()
 8  
 9 channel.queue_declare(queue=‘task_queue‘, durable=True)
10  
11 message = ‘ ‘.join(sys.argv[1:]) or "Hello World!"
12 channel.basic_publish(exchange=‘‘,
13                       routing_key=‘task_queue‘,
14                       body=message,
15                       properties=pika.BasicProperties(
16                          delivery_mode = 2, # make message persistent
17                       ))
18 print(" [x] Sent %r" % message)
19 connection.close()

消费者端

 1 #!/usr/bin/env python
 2 import pika
 3 import time
 4  
 5 connection = pika.BlockingConnection(pika.ConnectionParameters(
 6         host=‘localhost‘))
 7 channel = connection.channel()
 8  
 9 channel.queue_declare(queue=‘task_queue‘, durable=True)
10 print(‘ [*] Waiting for messages. To exit press CTRL+C‘)
11  
12 def callback(ch, method, properties, body):
13     print(" [x] Received %r" % body)
14     time.sleep(body.count(b‘.‘))
15     print(" [x] Done")
16     ch.basic_ack(delivery_tag = method.delivery_tag)
17  
18 channel.basic_qos(prefetch_count=1)
19 channel.basic_consume(callback,
20                       queue=‘task_queue‘)
21  
22 channel.start_consuming()