Windows docker-compose Kafka集群排坑

Posted 2022-05-25 nirvana · rebirth

Windows中，使用VMWare进行配置没有过多问题，但是当使用Windows下的docker时，Kafka的监听配置就是个大坑。
核心问题在于：

1. ADVERTISED_LISTENERS 和 LISTENERS 的理解
2. host.docker.internal 的理解

Kafka 的 ADVERTISED_LISTENERS 配置是保存的字符串，客户端请求的时候会返回这个配置信息。

---

docker-compose.yml

version: '3.8'

networks:
  kafka-cluster-net:
    driver: bridge


services:
  zookeeper:
    image: zookeeper:3.8
    container_name: kafka-cluster_zookeeper
    environment:
      - ALLOW_NONE_AUTHENTICATION=yes
    networks:
      - kafka-cluster-net

  kafka1:
    image: bitnami/kafka
    container_name: kafka1
    environment:
      - TZ=Asia/Shanghai
      - ALLOW_NONE_AUTHENTICATION=yes
      - ALLOW_PLAINTEXT_LISTENER=yes # 出于安全原因，Bitnami Apache Kafka docker 映像禁用了 PLAINTEXT 侦听器。
      - KAFKA_BROKER_ID=1 # 指定BrokerId, 如果不指定, 将会自己生成
      - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9093  # Kafka真正bind的地址
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://host.docker.internal:9093 # 暴露给外部的listeners, 默认是LISTENERS, 只能是除0.0.0.0外的合法ip或域名
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181/kafka # Zookeeper连接地址
      - KAFKA_ZOOKEEPER_TLS_VERIFY_HOSTNAME=false
      - KAFKA_TLS_CLIENT_AUTH=none
    privileged: true
    ports:
      - "9093:9093"
    networks:
      - kafka-cluster-net
    depends_on:
      - zookeeper

  kafka2:
    image: bitnami/kafka
    container_name: kafka2
    environment:
      - TZ=Asia/Shanghai
      - ALLOW_NONE_AUTHENTICATION=yes
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_BROKER_ID=2
      - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9094
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://host.docker.internal:9094
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181/kafka
      - KAFKA_ZOOKEEPER_TLS_VERIFY_HOSTNAME=false
      - KAFKA_TLS_CLIENT_AUTH=none
    privileged: true
    ports:
      - "9094:9094"
    networks:
      - kafka-cluster-net
    depends_on:
      - zookeeper

  kafka3:
    image: bitnami/kafka
    container_name: kafka3
    environment:
      - TZ=Asia/Shanghai
      - ALLOW_NONE_AUTHENTICATION=yes
      - ALLOW_PLAINTEXT_LISTENER=yes
      - KAFKA_BROKER_ID=3
      - KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9095
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://host.docker.internal:9095
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181/kafka
      - KAFKA_ZOOKEEPER_TLS_VERIFY_HOSTNAME=false
      - KAFKA_TLS_CLIENT_AUTH=none
    privileged: true
    ports:
      - "9095:9095"
    networks:
      - kafka-cluster-net
    depends_on:
      - zookeeper

---

资料

• Docker Hub bitnami/kafka
• 从Docker容器中访问宿主机网络
• kafka listeners 和 advertised.listeners 的应用