kafka_2.12集群布署
Posted ~~~~~~~~~~~~~~
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了kafka_2.12集群布署相关的知识,希望对你有一定的参考价值。
下载kafka:
https://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.5.0/kafka_2.12-2.5.0.tgz
配置JDK环境:
echo "export JAVA_HOME=/usr/local/jdk" >> /etc/profile echo "export CLASSPATH=\\$JAVA_HOME/lib:\\$CLASSPATH" >> /etc/profile echo "export PATH=\\$JAVA_HOME/bin:\\$PATH" >> /etc/profile source /etc/profile |
tar -xf kafka_2.12-2.5.0.tgz
mv kafka_2.12-2.5.0 /usr/local/kafka
cd /usr/local/kafka/
1、zookeeper集群配置
mkdir -p /data/zookeeper
修改各节点zookeeper.properties配置文件如下:
192.168.199.140
创建集群ID文件:echo 1 > /data/zookeeper/myid
dataDir=/data/zookeeper clientPort=2181 maxClientCnxns=0 admin.enableServer=true admin.serverPort=8080 initLimit=5 syncLimit=2 server.1=192.168.199.140:2888:3888 server.2=192.168.199.141:2888:3888 server.3=192.168.199.142:2888:3888 |
192.168.199.141
创建集群ID文件:echo 2 > /data/zookeeper/myid
dataDir=/data/zookeeper clientPort=2181 maxClientCnxns=0 admin.enableServer=true admin.serverPort=8080 initLimit=5 syncLimit=2 server.1=192.168.199.140:2888:3888 server.2=192.168.199.141:2888:3888 server.3=192.168.199.142:2888:3888 |
192.168.199.142
创建集群ID文件:echo 3 > /data/zookeeper/myid
dataDir=/data/zookeeper clientPort=2181 maxClientCnxns=0 admin.enableServer=true admin.serverPort=8080 initLimit=5 syncLimit=2 server.1=192.168.199.140:2888:3888 server.2=192.168.199.141:2888:3888 server.3=192.168.199.142:2888:3888 |
配置释义:
initLimit:允许follower与leader连接和同步的时间,如果ZooKeeper管理的数据量很大,可以根据需要增加这个值。 syncLimit:允许follower与leader同步的时间,如果follower远远落后于leader,他们将被丢弃。 maxClientCnxns:限制单个客户端(由IP地址标识)对ZooKeeper集成中的单个成员的并发连接数(在套接字级别)。这是用来防止某些类型的DoS攻击,包括文件描述符耗尽。默认值是60。将此值设置为0完全删除了并发连接的限制。 admin.enableServer:是否开启管理服务 admin.serverPort=8080:管理服务开放端口 clientPort:监听客户端连接的端口 server.id=host:port:port 对于每台服务器,您首先需要指定一个仲裁端口,然后为ZooKeeper leader选举指定一个专用端口 |
更多配置参数参考官网:
https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_zkMulitServerSetup
各节点启动zookeeper:
bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
集群管理命令:
url示例:/commands/[command name]
更多命令参考官网:
https://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_configuration
查看zookeeper集群服务状态:
http://192.168.199.142:8080/commands/stat
或
curl http://localhost:8080/commands/stat
由此查看到目前192.168.199.142这台是leader节点。
2、kafka集群配置(server.properties)
192.168.199.140
broker.id=0 listeners=PLAINTEXT://192.168.199.140:9092 advertised.listeners=PLAINTEXT://192.168.199.140:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=2 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.199.140:2181,192.168.199.141:2181,192.168.199.142:2181 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 |
192.168.199.141
broker.id=1 listeners=PLAINTEXT://192.168.199.141:9092 advertised.listeners=PLAINTEXT://192.168.199.141:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=2 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.199.140:2181,192.168.199.141:2181,192.168.199.142:2181 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 |
192.168.199.142
broker.id=2 listeners=PLAINTEXT://192.168.199.142:9092 advertised.listeners=PLAINTEXT://192.168.199.142:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/tmp/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=2 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.199.140:2181,192.168.199.141:2181,192.168.199.142:2181 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 |
参数参考官网:
http://kafka.apache.org/documentation/#connect_configuring
各节点启动kafka:
bin/kafka-server-start.sh -daemon config/server.properties
创建topic:
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server 192.168.199.140:9092
查看topic详细信息:
bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server 192.168.199.140:9092
模拟生产者写消息:
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server 192.168.199.140:9092
This is my first event
This is my second event
模拟消费者读消息:
$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server 192.168.199.140:9092
This is my first event
This is my second event
集群部署完成。
以上是关于kafka_2.12集群布署的主要内容,如果未能解决你的问题,请参考以下文章