Kafka动态调整topic副本因子replication-factor

Posted 2023-03-19

参考技术A

实际项目中我们可能在创建topic时没有设置好正确的replication-factor，导致kafka集群虽然是高可用的，但是该topic在有broker宕机时，可能发生无法使用的情况。topic一旦使用又不能轻易删除重建，因此动态增加副本因子就成为最终的选择。

原因分析：

假设我们有3个kafka broker分别brokerA、brokerB、brokerC.

如何动态给已经创建的topic添加replication-factor？
可能很多人想使用kafka-topics.sh脚本，那么事情情况如何了？

截图

可以看出kafka-topics.sh不能用来增加副本因子replication-factor。实际应该使用kafka bin目录下面的kafka-reassign-partitions.sh。
a, 首先我们配置topic的副本，保存为json文件（）
例如， 我们想把yqtopic01的部分设置为3，（我的kafka集群有3个broker，id分别为0,1,2）， json文件名称为increase-replication-factor.json

"version":1,
"partitions":[
"topic":"yqtopic01","partition":0,"replicas":[0,1,2],
"topic":"yqtopic01","partition":1,"replicas":[0,1,2],
"topic":"yqtopic01","partition":2,"replicas":[0,1,2]
]
b, 然后执行脚本
./kafka-reassign-partitions.sh -zookeeper 127.0.0.1:2181 --reassignment-json-file increase-replication-factor.json --execute

kafka-reassign-partitions.sh执行截图

我们可以通过执行
kafka-topics.sh --describe --zookeeper localhost:2181 --topic yqtopic01查看现在该topic的副本因子。

总结
所有文档官方文档最权威。 https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor

摘录如下：
Increasing replication factor
Increasing the replication factor of an existing partition is easy. Just specify the extra replicas in the custom reassignment json file and use it with the --execute option to increase the replication factor of the specified partitions.

For instance, the following example increases the replication factor of partition 0 of topic foo from 1 to 3. Before increasing the replication factor, the partition\'s only replica existed on broker 5. As part of increasing the replication factor, we will add more replicas on brokers 6 and 7.

The first step is to hand craft the custom reassignment plan in a json file:

Then, use the json file with the --execute option to start the reassignment process:

The --verify option can be used with the tool to check the status of the partition reassignment. Note that the same increase-replication-factor.json (used with the --execute option) should be used with the --verify option:

You can also verify the increase in replication factor with the kafka-topics tool: