连接到在 docker 容器中运行的 Kafka 时出错

Posted 2023-02-27

【中文标题】连接到在 docker 容器中运行的 Kafka 时出错

【英文标题】：Error connecting to Kafka running in docker container

【发布时间】：2021-07-21 22:18:44

【问题描述】：

我已经为部署到Wildfly 的EAR 的lib 目录中捆绑了基于spring boot 的库配置了以下Kafka 属性。我可以通过从类路径 (WEB-INF/classes) 加载 porperty 文件来成功启动 spring 组件

spring.autoconfigure.exclude=org.springframework.boot.autoconfigure.gson.GsonAutoConfiguration,org.springframework.boot.autoconfigure.jms.artemis.ArtemisAutoConfiguration,\
org.springframework.boot.autoconfigure.data.web.SpringDataWebAutoConfiguration
spring.kafka.admin.client-id=iris-admin-local
spring.kafka.producer.client-id=iris-producer-local
spring.kafka.producer.retries=3
spring.kafka.producer.properties.max.block.ms=2000
spring.kafka.producer.bootstrap-servers=127.0.0.1:19092
spring.kafka.producer.key-serializer=org.apache.kafka.common.serialization.StringSerializer
spring.kafka.producer.value-serializer=org.springframework.kafka.support.serializer.JsonSerializer
foo.app.kafka.executor.core-pool-size=10
foo.app.kafka.executor.max-pool-size=500
foo.app.kafka.executor.queue-capacity=1000

我通过docker compose运行Kafka和zookeeper，容器分别映射到主机端口12181和19092。发布失败并出现错误

19:37:42,914 ERROR [org.springframework.kafka.support.LoggingProducerListener] (swiftalker-3) Exception thrown when sending a message with key='543507' and payload='com.foo.app.kanban.defect.entity.KanbanDefect@84b13' to topic alm_swift-alm:: org.apache.kafka.common.errors.TimeoutException: Topic alm_swift-alm not present in metadata after 2000 ms.

19:37:43,124 WARN  [org.apache.kafka.clients.NetworkClient] (kafka-producer-network-thread | iris-producer-local-1) [Producer clientId=iris-producer-local-1] Error connecting to node 6be446692a1f:9092 (id: 1001 rack: null): java.net.UnknownHostException: 6be446692a1f
    at java.net.InetAddress.getAllByName0(InetAddress.java:1281)
    at java.net.InetAddress.getAllByName(InetAddress.java:1193)
    at java.net.InetAddress.getAllByName(InetAddress.java:1127)
    at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:110)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:403)
    at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:363)
    at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:151)
    at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:955)
    at org.apache.kafka.clients.NetworkClient.access$600(NetworkClient.java:73)
    at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1128)
    at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1016)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:547)
    at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:324)
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
    at java.lang.Thread.run(Thread.java:748)

现在这是在提供 spring.kafka.producer.bootstrap-servers=127.0.0.1:19092 属性之后。有趣的是

CONTAINER ID   NAMES                PORTS                                                                          CREATED          STATUS
2133c81ed51d   mongo                0.0.0.0:23556->27017/tcp, 0.0.0.0:23557->27018/tcp, 0.0.0.0:23558->27019/tcp   29 minutes ago   Up 29 minutes
f18b86d8739e   kafka-ui             0.0.0.0:18080->8080/tcp                                                        29 minutes ago   Up 29 minutes
6be446692a1f   kafka                0.0.0.0:19092->9092/tcp                                                        29 minutes ago   Up 29 minutes
873304e1e6a0   zookeeper            2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp                                         29 minutes ago   Up 29 minutes

Wildfly 服务器错误日志显示应用实际上是通过其容器 ID 连接到 docker 容器，即

6be446692a1f   kafka                0.0.0.0:19092->9092/tcp

来自docker ps -a 输出和

Error connecting to node 6be446692a1f:9092 (id: 1001 rack: null): java.net.UnknownHostException: 6be446692a1f

尽管配置属性通过 localhost 和映射端口 19092 引用服务器，但我对 Spring Boot 代码感到困惑，它设法在其 ID 和默认端口上找到一个 docker 容器，然后尝试连接到它？我该如何解决这个问题？

更新：docker compose

version: '3'

networks:
  app-tier:
    driver: bridge

services:
  zookeeper:
    image: 'docker.io/bitnami/zookeeper:3-debian-10'
    container_name: 'zookeeper'
    networks:
      - app-tier
    volumes:
      - 'zookeeper_data:/bitnami'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'docker.io/bitnami/kafka:2-debian-10'
    container_name: 'kafka'
    ports:
      - 19092:9092
    networks:
      - app-tier
    volumes:
      - 'kafka_data:/bitnami'
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper
  database:
    image: 'mongo'
    container_name: 'mongo'
    environment:
      - MONGO_INITDB_DATABASE='swiftalk_db'
    networks:
      - app-tier
    ports:
      - 23556-23558:27017-27019
    depends_on:
      - kafka
  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    ports:
      - 18080:8080
    networks:
      - app-tier
    volumes: 
      - 'mongo_data:/data/db'
    depends_on:
      - kafka
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:9092
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local
  mongo_data:
    driver: local

【参考方案1】：

您尚未共享您的 Docker Compose，因此我无法为您提供具体的修复方法，但本质上您需要正确配置广告侦听器。这是代理提供给客户端的值，告诉它在进行后续连接时在哪里找到它。

详情：https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/

【讨论】：

添加了docker-compose.yml；我没有正确理解我们所说的广告听众的意思

我知道我错过了KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:19092

阅读博客 :)