How to Support More Queues in RocketMQ?

Posted 2021-04-24 麦田里的守望者

总结

Kafka is a distributed streaming platform, which was born from logging aggregation cases. It does not need too high concurrency. In some large scale cases in alibaba, we found that the original model has been unable to meet our actual needs. So, we developed a messaging middleware, named RocketMQ, which can handle a broad set of use cases, ranging from traditional publish/subscribe scenario to demandingly high volume realtime transaction system that tolerates no message loss. Now, in alibaba, RocketMQ clusters process more than 500 billion events every day, provide services for more than 3000 core applications.

---

Kafka是一个分布式日志流平台，它底层是由日志聚合生成的。Kafka并不能支持特别高的并发。在Alibaba一些巨大规模案例中，我们发现原始的Kafka模型满足不了实际的需求。因此，我们开发了一个消息系统中间件（RocketMQ）,该中间件可以处理广泛的应用场景。从传统的pub/sub场景到海量、近实时、zero-loss容错事物系统。现在在Alibaba，RocketMQ集群每天处理5000亿events，为3000个核心应用提供服务。

Partition design in kafka

• 1.Producer parallelism of writing is bounded by the number of partitions.

• 2.The degree of consumer consumption parallelism, is also bounded by the number of partitions being consumed. Assuming that the number of partitions is 20, the maximum number of concurrent consuming consumers is 20.

• 3.Each topic consists of a fixed number of partitions. Partition number determines the maximum number of topics that single broker may have without significantly affecting performance.

More details please refer to here. http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/

---

• 1.生产者的写并行度是和分区数目严格绑定。

• 2.消费者消费并行度，从某种程度上也受限于分区。设想一个topic有20个分区，那么最大的消费并发数也只能是20个消费者同时消费。

• 3.每一个topic包含固定数目的分区，分区数目决定了topic最大的分区量，但是针对于单个broker来说性能上并没有重大的提升。

更多细节请参考: http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/

Why Kafka can’t support more partitions

• 1.Each partition stores the whole message data. Although each partition is orderly written to the disk, as number of concurrently writing partitions increases, writing become random in the perspective of operating system.

• 2.Due to the scattered data files, it is difficult to use the Linux IO Group Commit mechanism.

---

• 1.每一个分区存储整个message数据，虽然每一个分区顺序将数据写入磁盘，但是由于对分区写并发不断的提升，对于操作系统而言却是随机的。

• 2.由于数据文件分散，很难使用Linux IO的Group Commit机制。

How to support more partition in RocketMQ?

• 1.All message data are stored in commit log files. All writes are completely sequential whilst reads are random.

• 2.ConsumeQueue stores the actual user consumption location information, which are also flushed to disk in sequential manner.

---

• 1.所有的messsage数据都存储在commit log文件中，所有的写都是顺序的的，但是读操作是随机的。

• 2.ConsumeQueue 存储实际的消费位置信息，这些信息也会也是顺序方式flush到磁盘上.

producer

• 1.Each consume queue is lightweight and contains limited amount of meta data.

• 2.Access to disk is totally sequential, which avoids disk lock contention, and will not incur high disk IO wait when a large number of queues has been created.

---

• 1.每一个消费队列是非常轻量级和一些有限的meta数据

• 2.对于磁盘的访问都是全局有序的，这样可以避免磁盘锁竞争以及不会在创建大量queues的时候造成磁盘IO的等待。

consumer

• 1.Message consumption will first read consume queue, then commit log. This process brings in certain cost in worst cases.

• 2.Commit log and consume queues need to be logically consistent, which introduces extra complexities to programming model.

---

• 1.消费消息的时候需要先去读consume queue然后读取commit log.该种处理方式在最糟糕的时候会带来一定的成本。

• 2.Commit log 和 consume queues需要保持逻辑的一致性，这一点会在编程过程中带来一些复杂性。

DesignMotivation(设计动机)

• 1.Random read. Read as much as possible to increase the page cache hit rate, and reduce read IO operations. So large memory is still preferable. If massive messages are accumulated, would the read performance degrade badly? The answer is negative, reasons are as follows:Even if size of the message is only 1KB, the system will read more data in advance, see PAGECACHE prefetch for reference. This means for the sequel data read, it is access to main memory that will be carried out instead of slow disk IO read.

• 2.Given ConsumeQueue stores fixed-size metadata only, which is mainly used to record consuming progress, random read is well supported. Taking advantage of page cache prefetch, accessing ConsumeQueue is as efficiently fast as accessing main memory, even if it’s in the case of massive message accumulation. As a result，ConsumeQueue will NOT bring in noticeable penalty to the read performance.

• 3.CommitLog stores virtually all information, including the message data. Similar to redo log of relational database, consume queues, message key indexes and all other required data can be completely recovered as long as commit log exists.

---

• 1.随机读取。尽可能多地阅读以提高页面缓存命中率，并减少读取IO操作。因此，大容量存储器仍然是优选的。如果积累了大量的信息，阅读性能会下降吗？答案是否定的，原因如下：即使消息的大小只有1KB，系统将提前读取更多的数据，参见PACECACH预取以供参考。这意味着对于后继数据读取，它是访问将要执行的主存储器而不是慢速磁盘IO Read。

• 2.使用ConsumeQueue仅仅存储固定大小的元数据，支持随机读取，主要用于记录消费进度。利用页面缓存预读取，访问ConsumeQueue就像访问内存一样快，即使在大量消息累积的情况下也是如此。因此，ConsumeQueue不会明显的损害阅读性能。

• 3.CommitLog实际上存储所有信息，包括消息数据。与关系数据库的redo log类似，只要存在commit log，就可以完全恢复consume queues、message key indexes和所有其他必需的数据。