kafka迁移topic

Posted 2020-11-07 青木

1. 准备移动

• 这里假设要移动的topic名称为：topicA、topicB

vim topics-to-move.json

{"topics": [{"topic": "topicA"},
            {"topic": "topicB"}],
"version":1
}

• 将上述 topic 移动到 broker 3,4,5上，用 generate 命令生成 partition 分配 json 串，如下：

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 --topics-to-move-json-file topics-to-move.json --broker-list "3,4,5" --generate

• 将看到如下：

Current partition replica assignment
{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[0,2]},
{"topic":"topicA","partition":1,"replicas":[1,0]},
{"topic":"topicB","partition":1,"replicas":[0,1]},
{"topic":"topicA","partition":0,"replicas":[2,1]},
{"topic":"topicB","partition":0,"replicas":[2,0]},
{"topic":"topicB","partition":2,"replicas":[1,2]}]}

Proposed partition reassignment configuration

{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}

2. 执行移动

• 将 Proposed partition reassignment configuration 下方的数据写入 expand-cluster-reassignment.json

vim expand-cluster-reassignment.json

{"version":1,"partitions":[{"topic":"topicA","partition":2,"replicas":[3,4]},
{"topic":"topicA","partition":1,"replicas":[5,3]},
{"topic":"topicB","partition":1,"replicas":[3,4]},
{"topic":"topicA","partition":0,"replicas":[4,5]},
{"topic":"topicB","partition":0,"replicas":[5,3]},
{"topic":"topicB","partition":2,"replicas":[4,5]}]}

• 用 execute 命令开始正式执行，将会把上述2个topic按 expand-cluster-reassignment 中的设置进行分配，该过程可能会影响其它topic的读取（延时）

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--reassignment-json-file expand-cluster-reassignment.json --execute

3. 查看状态

• 查询执行状态

bin/kafka-reassign-partitions.sh --zookeeper zk1:2181 
--reassignment-json-file expand-cluster-reassignment.json --verify

• 主要有几种状态

Reassignment of partition [topicA,4] is still in progress # 转移中
Reassignment of partition [topicB,2] completed successfully # 转移结束

• 如果你看到如下方的ERROR信息，没关系，不要关闭节点，过会再看看就正常了

ERROR: Assigned replicas (0,5,1,2,3,4) don‘t match the list of replicas for reassignment (5,3,4) for partition [topicA,3]
Reassignment of partition [topicA,3] failed