kafka集群搭建

Posted 2022-03-02 健康平安的活着

一 集群规划阐述

192.168.152.128  master 192.168.152.129  slaver01 192.168.152.130  slaver02 安装包存放的目录：/ bigdata-software 安装程序存放的目录：/export/servers 三台机器执行以下命令创建统一文件目录 mkdir -p /export/servers mkdir -p /export/ bigdata-software

二 kafka集群搭建

在操作kafka操作需要，先启动zk集群，然后再每个节点去启动kafka节点。kafka搭建工作首先需要在：

https://blog.csdn.net/u011066470/article/details/122796425

这篇文章操作之后进行

2.1 前期准备工作

1.所有的机器均安装了jdk

2.zk的集群：保证三台机器的zk服务都正常启动，且正常运行，查看zk的运行状态，保证有一台zk的服务状态为leader，且两台为follower即可。

2.2  kafka的下载地址

http://archive.apache.org/dist/kafka/0.10.0.0/kafka_2.11-0.10.0.0.tgz

2.3 在master节点上kafka的解压和安装

将下载好的安装包上传到master服务器的/root/bigdata-software路径下，

然后进行解压kafka的压缩包，执行以下命令进行解压安装包：

[root@localhost ~]# cd bigdata-software/ [root@localhost bigdata-software]# ls jdk-8u171-linux-x64.tar.gz  kafka_2.11-0.10.0.0.tgz  zookeeper-3.4.8.tar.gz [root@localhost bigdata-software]# pwd /root/bigdata-software [root@localhost bigdata-software]# tar -zxvf kafka_2.11-0.10.0.0.tgz  -C /root/export/servers/

2.3.1 在master节点上新建kafka的log文件

在kafka的安装目录下，新建一个logs目录，如下：

[root@localhost kafka_2.11-0.10.0.0]# mkdir -p logs [root@localhost kafka_2.11-0.10.0.0]# ls bin  config  libs  LICENSE  logs  NOTICE  site-docs

2.3.2 在master节点上修改kafka的配置文件

在master节点下执行以下命令进入到kafka的配置文件目录，修改配置文件

cd    /root/export/servers/kafka_2.11-0.10.0.0/config [root@localhost config]# vi server.properties #修改的内容为如下：其中标红为本次修改的内容

broker.id=0 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/root/export/servers/kafka_2.11-0.10.0.0/logs num.partitions=2 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.flush.interval.messages=10000 log.flush.interval.ms=1000 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.152.136:2181,192.168.152.138:2181,192.168.152.140:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 delete.topic.enable=true host.name=192.168.152.136

2.3.3 在master节点上复制kfaka配置到slaver01节点上

root@localhost servers]# scp -r kafka_2.11-0.10.0.0/   root@192.168.152.138:/root/export/servers/ root@192.168.152.129's password: LICENSE                                                                                                                                              100%   28KB   2.8MB/s   00:00     NOTICE                                                                                                                                               100%  336    91.8KB/s   00:00     connect-distributed.sh                                                                                                                               100% 1052   435.7KB/s   00:00     connect-standalone.sh                                                                                                                                100% 1051   422.7KB/s   00:00     kafka-acls.sh

2.3.4 在slaver01节点上修改配置

在slaver01节点下执行以下命令进入到kafka的配置文件目录，修改配置文件

cd    /root/export/servers/kafka_2.11-0.10.0.0/config [root@localhost config]# vi server.properties #修改的内容为如下：其中标红为本次修改的内容 broker.id=1 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/root/export/servers/kafka_2.11-0.10.0.0/logs num.partitions=2 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.flush.interval.messages=10000 log.flush.interval.ms=1000 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.152.136:2181,192.168.152.138:2181,192.168.152.140:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 delete.topic.enable=true host.name=192.168.152.138

2.3.5 在master节点上复制kafka配置到slaver02节点上

[root@localhost servers]# scp -r kafka_2.11-0.10.0.0/   root@192.168.152.140:/root/export/servers/ root@192.168.152.129's password:

2.3.6  在slaver02节点上修改配置文件

在slaver02节点下执行以下命令进入到kafka的配置文件目录，修改配置文件

broker.id=2 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/root/export/servers/kafka_2.11-0.10.0.0/logs num.partitions=2 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.flush.interval.messages=10000 log.flush.interval.ms=1000 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.152.136:2181,192.168.152.138:2181,192.168.152.140:2181 zookeeper.connection.timeout.ms=6000 group.initial.rebalance.delay.ms=0 delete.topic.enable=true host.name=192.168.152.140

2.4 kafka的启动

2.4.1 kafka的前提约束

注意事项：在kafka启动前，一定要让zookeeper启动起来。

2.4.2 在master节点上后台启动命令

在master节点执行以下命令将kafka进程启动在后台，通过jps命令来查看进程是否存在

[root@localhost kafka_2.11-0.10.0.0]# jps 8486 Jps 4216 QuorumPeerMain [root@localhost kafka_2.11-0.10.0.0]# nohup bin/kafka-server-start.sh config/server.properties 2>&1 & [1] 8502 nohup: ignoring input and appending output to ‘nohup.out’ [root@localhost kafka_2.11-0.10.0.0]# jps 8752 Jps 8502 Kafka 4216 QuorumPeerMain [root@localhost kafka_2.11-0.10.0.0]#

2.4.3 在master节点上后台关闭命令

bin/kafka-server-stop.sh

2.4.4 在slaver01节点上后台启动命令

在slaver01节点执行以下命令将kafka进程启动在后台，通过jps命令来查看进程是否存在

[root@localhost kafka_2.11-0.10.0.0]# nohup bin/kafka-server-start.sh config/server.properties 2>&1 & [1] 2904 nohup: ignoring input and appending output to ‘nohup.out’ [root@localhost kafka_2.11-0.10.0.0]# jps 2904 Kafka 2763 QuorumPeerMain 3118 Jps [root@localhost kafka_2.11-0.10.0.0]# ls bin  config  libs  LICENSE  logs  nohup.out  NOTICE  site-docs

2.4.5 在slaver01节点上后台关闭命令

bin/kafka-server-stop.sh

2.4.6 在slaver02节点上后台启动命令

在slaver02节点执行以下命令将kafka进程启动在后台，通过jps命令来查看进程是否存在

[root@localhost kafka_2.11-0.10.0.0]# nohup bin/kafka-server-start.sh config/server.properties 2>&1 & [1] 2904 nohup: ignoring input and appending output to ‘nohup.out’ [root@localhost kafka_2.11-0.10.0.0]# jps 2904 Kafka 2763 QuorumPeerMain 3118 Jps [root@localhost kafka_2.11-0.10.0.0]# ls bin  config  libs  LICENSE  logs  nohup.out  NOTICE  site-docs

2.4.7  在slaver02节点上后台关闭命令

bin/kafka-server-stop.sh

2.4.8 查看每个节点的nohup.out的输出日志

#master节点：

[root@bogon kafka_2.12-2.1.0]# tail -f nohup.out

 

#slaver01节点：

[root@bogon kafka_2.12-2.1.0]# tail -f nohup.out

#slaver02节点：

[root@bogon kafka_2.12-2.1.0]# tail -f nohup.out

可以看到：这3个节点都成功start，没有报错，ok，到此kafka的集群搭建完成。