Kafka安装部署

Posted 2020-12-14 sunshinea121

Kafka安装部署

Kafka依赖zookeeper，默认kafka有自带的zookeeper，但是一般情况使用自建的好一些

1、    安装zookeeper

1)      创建目录

mkdir /data/kafka/zookeeper/{log,data} -p

2)      下载zookeeper安装包

cd /data/kafka && wget https://downloads.apache.org/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz

3)      解压

tar xf zookeeper-3.4.14.tar.gz && cd zookeeper-3.4.14/conf

4)      编辑配置文件

mv zoo_sample.cfg zoo.cfg

配置文件内容

cat zoo.cfg

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/data/kafka/zookeeper/data

dataLogDir=/data/kafka/zookeeper/log

clientPort=2181

server.1=172.17.42.116:2888:3888

server.2=172.17.42.118:2888:3888

server.3=172.17.51.173:2888:3888

5)      写入节点ID

echo 1 > /data/kafka/zookeeper/data/myid

6)      配置zookeeper的日志切割

修改conf/log4j.properties

zookeeper.root.logger=INFO,ROLLINGFILE

zookeeper.log.dir=/data/kafka/zookeeper/logs

log4j.appender.ROLLINGFILE=org.apache.log4j.DailyRollingFileAppender

#log4j.appender.ROLLINGFILE.MaxFileSize=10MB 

log4j.appender.ROLLINGFILE.DatePattern=‘.‘yyyy-MM-dd

7)      修改bin/zkEnv.sh

if [ "x${ZOO_LOG_DIR}" = "x" ]

then

ZOO_LOG_DIR="/data/kakfa/zookeeper/logs"     //设置zookeeper日志目录

fi

 

if [ "x${ZOO_LOG4J_PROP}" = "x" ]

then

    ZOO_LOG4J_PROP="INFO,ROLLINGFILE"    //将CONSOLE修改为ROLLINGFILE

Fi

8)      修改zookeeper日志文件名

vim bin/zkServer.sh

_ZOO_DAEMON_OUT="$ZOO_LOG_DIR/zookeeper.log"   //将zookeeper.out修改为zookeeper.log

ZOOMAIN="-Dzookeeper.4lw.commands.whitelist=* ${ZOOMAIN}"   添加至78行

9)      传送到其他两台机器，并更改myid

rsync -avz /data/kafka/zookeeper /data/kafka/zookeeper-3.4.14 172.117.42.118:/data/kafka

rsync -avz /data/kafka/zookeeper /data/kafka/zookeeper-3.4.14 172.117.51.173:/data/kafka

10) 更改节点2的myid值

echo 2 > /data/kafka/zookeeper/data/myid

11) 更改节点3的myid值

echo 3 > /data/kafka/zookeeper/data/myid

12) Zookeeper启动，停止，重启

/data/kafka/zookeeper-3.4.14/bin/zkServer.sh start

/data/kafka/zookeeper-3.4.14/bin/zkServer.sh stop

/data/kafka/zookeeper-3.4.14/bin/zkServer.sh restart

2、    Kafka部署

1)      下载kafka

wget https://archive.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz

2)      编辑配置文件

cd /data/kafka/kafka_2.12-2.3.0/config

vim server.properties

broker.id=1                         #id每台机器不能一样

delete.topic.enable=true

listeners=PLAINTEXT://192.168.233.167:9092  #填写本机IP地址，填写主机名要可以解析

advertised.listeners=PLAINTEXT://192.168.233.167:9092

host.name=192.168.233.167

advertised.host.name=192.168.233.167

num.network.threads=3

num.io.threads=9

socket.send.buffer.bytes=102400

socket.receive.buffer.bytes=102400

socket.request.max.bytes=104857600

log.dirs=/data/kafka/kafka_2.12-2.3.0/logs   #日志目录

num.partitions=9

num.recovery.threads.per.data.dir=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

zookeeper.connect=192.168.233.167:2181,192.168.233.168:2181,192.168.233.169:2181#zookeeper集群地址

zookeeper.connection.timeout.ms=12000

auto.create.topics.enable=false

unclean.leader.election.enable=false

3)      配置节点2

broker.id=2                           #id每台机器不能一样

listeners=PLAINTEXT://192.168.233.168:9092  #填写本机IP地址，填写主机名要可以解析

advertised.listeners=PLAINTEXT://192.168.233.168:9092

host.name=192.168.233.168

advertised.host.name=192.168.233.168

4)      配置节点3

broker.id=3                           #id每台机器不能一样

listeners=PLAINTEXT://192.168.233.169:9092  #填写本机IP地址，填写主机名要可以解析

advertised.listeners=PLAINTEXT://192.168.233.169:9092

host.name=192.168.233.169

advertised.host.name=192.168.233.169

5)      启动kafka

/data/kafka/kafka_2.12-2.3.0/bin/kafka-server-start.sh -daemon /data/kafka/kafka_2.12-2.3.0/config/server.properties

6)      Kafka管理工具

Kafka管理工具用docker方式部署

                        i.              安装docker

yum install docker-ce

                      ii.              下载镜像

docker pull dockerkafka/kafka-manager

                    iii.              启动kafka-manager

docker run -d -p 9000:9000 -e ZK_HOSTS=172.17.42.117:2181,172.17.42.118:2181,172.17.51.173:2181 --name kafka-manager dockerkafka/kafka-manager