Debezium：无法在 Kafka-Connect Docker 容器上设置观察程序。获取相关 ID 为 2 的元数据时出错 ... LEADER_NOT_AVAILABLE

Posted 2023-03-10

技术标签:

【中文标题】Debezium：无法在 Kafka-Connect Docker 容器上设置观察程序。获取相关 ID 为 2 的元数据时出错 ... LEADER_NOT_AVAILABLE【英文标题】：Debezium: Unable to setup watcher on Kafka-Connect Docker container. Error while fetching metadata with correlation id 2 ... LEADER_NOT_AVAILABLE 【发布时间】：2020-07-21 04:42:08 【问题描述】：

问题：

如何设置观察者以在对数据库进行更改时实时观察 Kafka-Connect 流？我收到Error while fetching metadata with correlation id 2 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE。是什么导致了问题，我该如何解决？

上下文：

此问题与：Debezium How do I correctly register the SqlServer connector with Kafka Connect - connection refused 和 https://github.com/debezium/debezium-examples/blob/master/tutorial/README.md#using-sql-server

在 Windows 10 中，我使用以下命令在各自独立的 Docker 容器中设置了 Zookeeper、Kafka 和 Kafka-Connector：

docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.1
docker run -it --rm --name kafka -p 9092:9092 -e KAFKA_ADVERTISED_HOST_NAME: kafka --link zookeeper:zookeeper debezium/kafka:1.1
docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka debezium/connect:1.1

然后，我将 Kafka 连接器设置到在 Docker 容器之外的本地主机上运行的 SQL Server 实例。日志显示连接器可以连接到数据库并获取 CDC 数据。到目前为止，我还可以。

但是，当我尝试创建观察者时，使用以下命令：

docker run -it --rm --name watcher --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.1

我收到以下错误。

watch-topic -a -k dbhistory.Test.Posts
WARNING: Using default BROKER_ID=1, which is valid only for non-clustered installations.
Using ZOOKEEPER_CONNECT=172.17.0.2:2181
Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.5:9092
Using KAFKA_BROKER=172.17.0.3:9092
Contents of topic dbhistory.Test.Posts:
[2020-04-08 15:18:39,373] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 2 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:41:20,501] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:41:20,605] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:41:20,807] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:41:21,262] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:41:22,120] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:41:23,330] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:41:24,189] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:41:59,826] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:04,051] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:07,121] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:08,024] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:09,354] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11570 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:09,458] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11572 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:09,571] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11574 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:09,677] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11576 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:09,786] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11578 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:09,895] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11580 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,002] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11582 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,112] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11584 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,220] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11586 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,326] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11588 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,435] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11590 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,537] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11592 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,648] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11594 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,751] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11596 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 16:42:10,854] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Error while fetching metadata with correlation id 11598 : dbhistory.Test.Posts=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)
[2020-04-08 17:08:48,268] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2020-04-08 17:09:55,602] WARN [Consumer clientId=consumer-console-consumer-3848-1, groupId=console-consumer-3848] Connection to node 1 (/172.17.0.3:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

【问题讨论】：

你好，这两行是明显的矛盾 ``` Using KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://172.17.0.5:9092 Using KAFKA_BROKER=172.17.0.3:9092 ``` IP应该是一样的两种情况。 Kafka 容器的 IP 是多少？ -e KAFKA_ADVERTISED_HOST_NAME: kafka 也是正确的格式吗？我通常在设置环境变量时使用等号 @JiriPechanec 我在错误中看到了这一点，但我该如何解决？请注意，我使用的是帮助站点上提供给我的默认值。我的假设是它们应该开箱即用。您可以选择使用 Docker Compose 吗？它会让你的生活很多更轻松。如果可以接受，我可以根据 Docker Compose 提供答案。是的。对于概念证明。现在我正在设置它以在 Docker 中运行。要求是 Debezium 必须连接到机器上而不是 Docker 容器中的 SQL Server 实例。最终，它将在 Docker 之外运行。 【参考方案1】：

您可以尝试在 docker run 中添加 --network=host 一次吗？

【讨论】：

【参考方案2】：

问题很可能是由您定义环境变量的方式引起的。

替换：