Kafka— —副本（均衡负载）

Posted 2020-08-24 jht_newbie

创建一个副本数为3的topic

Now create a new topic with a replication factor of three:

> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

使用describe topics指令，查看副本在集群中每一个broker的分布情况。

Okay but now that we have a cluster how can we know which broker is doing what? To see that run the "describe topics" command:

> bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic
Topic:my-replicated-topic	PartitionCount:1	ReplicationFactor:3	Configs:
	Topic: my-replicated-topic	Partition: 0	Leader: 1	Replicas: 1,2,0	Isr: 1,2,0

  leader是负责该partition所有读和写的节点，每个节点随机选取一部分partition作为其leader。（为了保证较高的处理效率，消息的读写都是在固定的一个副本上完成。这个副本所在节点就是所谓的Leader，而其他副本所在节点则是Follower。而Follower则会定期地同步Leader上的数据。）

"leader" is the node responsible for all reads and writes for the given partition. 
Each node will be the leader for a randomly selected portion of the partitions.

  replicas是一个备份了该partition的节点的列表，不论他们是否是leader，也不管这些节点目前是否活着。

"replicas" is the list of nodes that replicate the log for this partition 
regardless of whether they are the leader or even if they are currently alive.

  isr是一个处于同步的副本节点集合。这是replicas的活着的并能与leader保持同步的节点的子集。（如果某个分区所在的leader服务器出了问题，不可用，kafka会从该分区的其他的副本中选择一个作为新的Leader。之后所有的读写就会转移到这个新的Leader上。现在的问题是应当选择哪个作为新的Leader。显然，只有那些跟Leader保持同步的Follower才应该被选作新的Leader。所以会从isr中选取新的Leader，通过ISR，kafka需要的冗余度较低，可以容忍的失败数比较高。假设某个topic有f+1个副本，kafka可以容忍f个服务器不可用。）

"isr" is the set of "in-sync" replicas. 
This is the subset of the replicas list that is currently alive and caught-up to the leader.

 

（参考kafka副本工具：https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-2.PreferredReplicaLeaderElectionTool）

 

  通过副本机制，每个partition可以有多个副本。一个副本的列表被称为分配副本。
  副本列表中的第一个被称为首选副本。当一个topic或分区被创建时，kafka保证了各个分区的首选副本均匀分布在集群的brokers上。
  在理想场景下，一个指定分区的leader应该是首选副本。

With replication, each partition can have multiple replicas. The list of replicas for a partition is called the "assigned replicas". 
The first replica in this list is the "preferred replica". When topic/partitions are created,
Kafka ensures that the "preferred replica" for the partitions across topics are equally distributed amongst the brokers in a cluster. 
In an ideal scenario, the leader for a given partition should be the "preferred replica".

  这些保证了，leader的负载在brokers直接最终是均衡的。
  然而，一段时间过后，leader的负载平衡可能被打破，例如broker停止工作。
  这个工具帮助恢复集群中各个brokers之间的负载均衡。
  工具使用主要有以下3步：

This guarantees that the leadership load across the brokers in a cluster are evenly balanced. 
However, over time the leadership load could get imbalanced due to broker shutdowns (caused by controlled shutdown, crashes, machine failures etc).
This tool helps to restore the leadership balance between the brokers in the cluster. 
A summary of the steps that the tool does is shown below -

  1.更新zookeeper的admin/prefered_replica_election目录下的list，该list记录了哪些topic的分区的leader需要被移动到首选副本。

1. The tool updates the zookeeper path "/admin/preferred_replica_election" with the list of topic partitions 
whose leader needs to be moved to the preferred replica.

  2.控制器监听以上的目录，当数据改变触发时，控制器从zookeeper读取topic的partition列表。

2. The controller listens to the path above.
 When a data change update is triggered, 
the controller reads the list of topic partitions from zookeeper.

  对于topic的每一个partition，控制器获取其首选副本（分配副本列表中的第一个副本）

3. For each topic partition, 
the controller gets the preferred replica
 (the first replica in the assigned replicas list). 

如果首选副本已经不是leader并且它在isr列表中，控制器发送一个请求给broker，控制首选副本称为该partition的leader。

If the preferred replica is not already the leader and it is present in the isr,
the controller issues a request to the broker 
that owns the preferred replica to become the leader for the partition.

  请注意，这个工具只更新zookeeper路径 。控制器异步地把一个partition的首选副本变成leader。

Note that the tool only updates the zookeeper path and exits.
 The controller moves the leader for a partition to the preferred replica asynchronously.

How to use the tool?

bin/kafka-preferred-replica-election.sh --zookeeper localhost:12913/kafka --path-to-json-file topicPartitionList.json

这个工具使用了zookeeper主机的强制性的列表和一个可选的列表（该列表以json文件形式指明了topic的分区）。如果没有指定list，这个工具查询zookeeper并获取集群所有topic的各个分区信息。工具用topic分区列表更新zookeeper路径“/admin/preferred_replica_election”后推出。

The tool takes a mandatory list of zookeeper hosts and an optional list of topic partitions provided as a json file. If the list is not provided, the tool queries zookeeper and 
gets all the topic partitions for the cluster. The tool exits after updating the zookeeper path "/admin/preferred_replica_election" with the topic partition list.

下面是json文件的例子（这是可选的。这个可以用来指定，移动特定的topic分区的leader到首选副本）

Example json file (This is optional. This can be specified to move the leader to the preferred replica for specific topic partitions)

{
 "partitions":
  [
    {"topic": "topic1", "partition": "0"},
    {"topic": "topic1", "partition": "1"},
    {"topic": "topic1", "partition": "2"},
 
    {"topic": "topic2", "partition": "0"},
    {"topic": "topic2", "partition": "1"},
  ]
}