Kafka 生产者连接到 localhost 而不是真实 IP

Posted 2023-03-10

【中文标题】Kafka 生产者连接到 localhost 而不是真实 IP

【英文标题】：Kafka producer is connecting to localhost instead of the real IP

【发布时间】：2020-05-09 02:00:01

【问题描述】：

请帮助我想知道为什么 kafka 生产者总是连接到本地主机但是代理 ip 不是本地主机。那么，有什么帮助吗？有什么想法吗？

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

import java.util.Properties;

public class ProducerDemo 

    public static void main(String[] args) throws Exception
        String bootstrapServers = "192.168.199.137:9092";

        Properties properties = new Properties();
        properties.setProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        properties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        properties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        // create the producer
        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);

        System.out.println("kafka IP   " + properties.getProperty(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG));


        // create producer record
        ProducerRecord<String, String> record = new ProducerRecord<String, String>("first_topic", "Hello world");


        // send data
        producer.send(record);
        producer.flush();
        producer.close();

    

这是 pom 内容

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>KafkaBeginnersCourse</artifactId>
    <version>1.0-SNAPSHOT</version>

    <packaging>jar</packaging>

    <dependencies>
        <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>2.4.0</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>slf4j-simple</artifactId>
            <version>1.7.28</version>
            <!--scope>test</scope-->
        </dependency>
    </dependencies>
</project>

这是来自输出控制台的列表

[kafka-producer-network-thread | producer-1] INFO org.apache.kafka.clients.Metadata - [Producer clientId=producer-1] Cluster ID: 0TPD87gWR0G18RLKk4gPow
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
[kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

【问题讨论】：

您能分享有关 Kafka 设置的详细信息吗？会不会是监听器配置问题？

我在 Centos 虚拟机上运行 kafka，我只是按照快速启动kafka.apache.org/quickstart 中的相同步骤进行操作，如果我打包应用程序并从虚拟机内部运行，则工作正常。但是，当我从 Windows 运行 Intellij 时，它不起作用。

我怀疑您需要更改 listeners 或添加 advertised.listeners 配置。您能否用网络配置和当前代理配置的详细描述更新问题。非常感谢。

【参考方案1】：

@cricket_007 已经说过，server.properties 文件中的advertised.listeners 设置似乎存在问题。让我试着详细解释一下。

当您的生产者尝试连接 Kafka 代理时，代理将广告主机名发送给客户端以供使用，然后生产者使用此地址连接到代理。所以正常的交流是这样的：

但在您的情况下，生产者和代理之间的通信是这样的：

P.S：您可以将图像中的公共 IP 假定为您的案例的私有 IP。

因此，您应该像这样在 server.properties 中设置广告侦听器：

advertised.listeners=PLAINTEXT://192.168.199.137:9092

图片参考：https://www.udemy.com/course/kafka-cluster-setup/

【参考方案2】：

bootstrap.servers 发现所有节点，您可以只提供一个地址，但首选更多。

您的客户端实际上连接到所有代理的已定义advertised.listeners 的返回列表。

听起来你被返回 localhost 作为那些听众。