Kubernetes-Kafka 无法在主题上写入消息

Posted

技术标签:

【中文标题】Kubernetes-Kafka 无法在主题上写入消息【英文标题】:Kubernetes-Kafka unable to write message on topic 【发布时间】:2019-06-04 06:12:18 【问题描述】:

我正在尝试编写有关 kafka 主题的数据,但遇到了一些错误。以下是我的配置和错误详情。

Kubernetes 服务:

kubectl get services
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kafka-service   NodePort    10.105.214.246   <none>        9092:30998/TCP               17m
kubernetes      ClusterIP   10.96.0.1        <none>        443/TCP                      4d
zoo1            ClusterIP   10.101.3.128     <none>        2181/TCP,2888/TCP,3888/TCP   20m

Kubernetes Pod:

kubectl get pods
NAME                                      READY   STATUS    RESTARTS   AGE
kafka-broker0-69c97b67f-4pmw9             1/1     Running   1          1m
zookeeper-deployment-1-796f9d9bcc-cr756   1/1     Running   0          20m

Kafka Docker 进程:

docker ps | grep kafka
f79cd0196083        wurstmeister/kafka@sha256:d04dafd2b308f26dbeed8454f67c321579c2818c1eff5e8f695e14a19b1d599b                         "start-kafka.sh"         About a minute ago   Up About a minute                       k8s_kafka_kafka-broker0-69c97b67f-4pmw9_default_a747d38a-0da6-11e9-bd84-fa163e7d3173_1
75393e9e25c1        k8s.gcr.io/pause-amd64:3.1                                                                                         "/pause"                 About a minute ago   Up About a minute                       k8s_POD_kafka-broker0-69c97b67f-4pmw9_default_a747d38a-0da6-11e9-bd84-fa163e7d3173_0

主题test在Kafka中创建成功如下图:

docker exec k8s_kafka_kafka-broker0-69c97b67f-4pmw9_default_a747d38a-0da6-11e9-bd84-fa163e7d3173_1 /opt/kafka_2.12-2.1.0/bin/kafka-topics.sh --list --zookeeper zoo1:2181
OR
docker exec k8s_kafka_kafka-broker0-69c97b67f-4pmw9_default_a747d38a-0da6-11e9-bd84-fa163e7d3173_1 /opt/kafka_2.12-2.1.0/bin/kafka-topics.sh --list --zookeeper 10.101.3.128:2181

上述命令的输出:

test

由于主题可用于在其上写入数据,因此我已使用主机 IP 10.225.36.98 或服务 IP 10.105.214.246 执行以下命令:

kubectl exec kafka-broker0-69c97b67f-4pmw9 -c kafka -i -t -- 
/opt/kafka_2.12-2.1.0/bin/kafka-console-producer.sh [ --broker-list 
10.225.36.98:30998  --topic test ]
>"k":"v"

但它们都不适合我并抛出以下异常:

[2019-01-01 09:26:52,215] ERROR Error when sending message to topic test with key: null, value: 9 bytes with error: 
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
>[2019-01-01 09:27:59,513] WARN [Producer clientId=console-producer] 
Connection to node -1 (/10.225.36.98:30998) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

当尝试使用主机名kafka 写入代理时:

kubectl exec kafka-broker0-69c97b67f-4pmw9 -c kafka -i -t -- /opt/kafka_2.12-2.1.0/bin/kafka-console-producer.sh [ --broker-list kafka:9092  --topic test ]

[2019-01-01 09:34:41,293] WARN Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
(org.apache.kafka.clients.ClientUtils)
org.apache.kafka.common.KafkaException: Failed to construct kafka producer

由于主机和服务 IP 不起作用,我尝试使用 pod IP,但收到 test=LEADER_NOT_AVAILABLE 错误。

kubectl exec kafka-broker0-69c97b67f-4pmw9 -c kafka -i -t -- /opt/kafka_2.12-2.1.0/bin/kafka-console-producer.sh [ --broker-list 172.17.0.7:9092  --topic test ]
>"k":"v"

[2019-01-01 09:52:30,733] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1 : test=LEADER_NOT_AVAILABLE (org.apache.kafka.clients.NetworkClient)

在谷歌搜索后,我找到了获取 Zookeeper 中可用代理列表的命令。所以我尝试从容器中运行它并陷入以下错误:

bash-4.4# ./opt/zookeeper/bin/zkCli.sh -server zoo1:2181 ls /brokers/ids
Connecting to zoo1:2181

来自 Zookeeper 的异常:

2019-01-01 09:18:05,215 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2019-01-01 09:18:05,219 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=zookeeper-deployment-1-796f9d9bcc-cr756
2019-01-01 09:18:05,220 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.8.0_151
2019-01-01 09:18:05,223 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2019-01-01 09:18:05,223 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8-openjdk/jre
2019-01-01 09:18:05,223 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/opt/zookeeper/bin/../build/classes:/opt/zookeeper/b
in/../build/lib/*.jar:/opt/zookeeper/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper/bin/../lib/slf4j-api-
1.6.1.jar:/opt/zookeeper/bin/../lib/netty-3.10.5.Final.jar:/opt/zookeeper/bin/../lib/log4j-
1.2.16.jar:/opt/zookeeper/bin/../lib/jline-0.9.94.jar:/opt/zookeeper/bin/../zookeeper-3.4.10.jar:/opt/zookeeper/bin/../src/java/lib/*.jar:/opt/zookeeper/bin/../conf:
2019-01-01 09:18:05,223 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/lib/jvm/java-1.8-
openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2019-01-01 09:18:05,223 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2019-01-01 09:18:05,224 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2019-01-01 09:18:05,224 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2019-01-01 09:18:05,224 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2019-01-01 09:18:05,224 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=3.10.0-693.11.6.el7.x86_64
2019-01-01 09:18:05,224 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2019-01-01 09:18:05,224 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2019-01-01 09:18:05,224 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/
2019-01-01 09:18:05,225 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=zoo1:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@25f38edc
2019-01-01 09:18:05,259 [myid:] - INFO  [main-SendThread(zoo1.default.svc.cluster.local:2181):ClientCnxn$SendThread@1032] - Opening socket connection to server zoo1.default.svc.cluster.local/10.101.3.128:2181. Will not attempt to authenticate using SASL (unknown error)
2019-01-01 09:18:35,280 [myid:] - WARN  [main-SendThread(zoo1.default.svc.cluster.local:2181):ClientCnxn$SendThread@1108] - Client session timed out, have not heard from server in 30027ms for sessionid 0x0
2019-01-01 09:18:35,282 [myid:] - INFO  [main-SendThread(zoo1.default.svc.cluster.local:2181):ClientCnxn$SendThread@1156] - Client session timed out, have not heard from server in 30027ms for sessionid 0x0, closing socket connection and attempting reconnect
Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /brokers/ids
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1532)
    at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1560)
    at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:731)
    at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:599)
    at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:362)
    at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:290)

我也尝试创建 LoadBalancer 类型的 Kafka 服务,但是,没有为服务分配 LoadBalancer IP。

解决此问题的参考资料:

    https://rmoff.net/2018/08/02/kafka-listeners-explained/ https://github.com/wurstmeister/kafka-docker/wiki/Connectivity#additional-listener-information https://github.com/kubernetes/contrib/issues/2891 https://dzone.com/articles/ultimate-guide-to-installing-kafka-docker-on-kuber https://github.com/wurstmeister/kafka-docker/issues/85

任何帮助将不胜感激。

【问题讨论】:

DNS resolution failed for kafka -- 听起来 CoreDNS 没有将其设置为该服务/pod 的主机名 我可以做些什么来设置主机名? CoreDNS 在我的设置中多次重启,这可能是什么原因? 【参考方案1】:

尝试使用以下命令向主题发送数据:

    docker exec k8s_kafka_kafka-broker0-69c97b67f-4pmw9_default_a747d38a-0da6-11e9-bd84-fa163e7d3173_1 
/opt/kafka_2.12-2.1.0/bin/kafka-console-producer.sh 
--broker-list kafka-service:30998 --topic test

【讨论】:

以上是关于Kubernetes-Kafka 无法在主题上写入消息的主要内容,如果未能解决你的问题,请参考以下文章

KafkaStreams在读取主题时写入磁盘

获取所有写入特定主题的 Kafka 源连接器

如何将火花流 DF 写入 Kafka 主题

无法使用 Xamarin Android/Uwp 在 USB 闪存驱动器中写入文件

在 iOS 9.2 上无法接收 FCM 主题消息

旧版 MFC 应用程序在具有 Aero 主题的 windows7 操作系统上无法正常运行,但在经典主题下表现良好