kafka集群的部署
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了kafka集群的部署相关的知识，希望对你有一定的参考价值。
1.zookeeper集群部署不在简介
zk对外提供2181端口 分别部署在三台机器上面，也是下面的三台
2.hosts文件一定要做 否则kafka系统启动不起来
[[email protected] bin]# cat /etc/hosts 192.168.56.11 linux-node1 192.168.56.12 linux-node2 192.168.56.13 linux-node3
3.linux-node1
[[email protected] ~]# ll total 202032 -rw-------. 1 root root 1011 Aug 26 18:00 anaconda-ks.cfg lrwxrwxrwx 1 root root 17 Oct 5 02:41 jdk -> /root/jdk1.7.0_79 drwxr-xr-x 8 10 143 4096 Apr 11 2015 jdk1.7.0_79 -rw-r--r-- 1 root root 153512879 Oct 5 02:39 jdk-7u79-linux-x64.tar.gz lrwxrwxrwx 1 root root 24 Oct 5 03:03 kafka01 -> /root/kafka_2.11-0.9.0.1 drwxr-xr-x 7 root root 94 Oct 5 03:18 kafka_2.11-0.9.0.1 -rw-r--r-- 1 root root 35650542 Oct 5 03:02 kafka_2.11-0.9.0.1.tgz lrwxrwxrwx 1 root root 21 Oct 5 02:43 zk01 -> /root/zookeeper-3.4.6 drwxr-xr-x 10 1000 1000 4096 Feb 20 2014 zookeeper-3.4.6 -rw-r--r-- 1 root root 17699306 Oct 5 02:43 zookeeper-3.4.6.tar.gz [[email protected] ~]#
linux-node2
[[email protected] ~]# ll total 202032 -rw-------. 1 root root 1011 Aug 26 18:00 anaconda-ks.cfg lrwxrwxrwx 1 root root 17 Oct 5 02:41 jdk -> /root/jdk1.7.0_79 drwxr-xr-x 8 10 143 4096 Apr 11 2015 jdk1.7.0_79 -rw-r--r-- 1 root root 153512879 Oct 5 02:39 jdk-7u79-linux-x64.tar.gz lrwxrwxrwx 1 root root 24 Oct 5 03:03 kafka02 -> /root/kafka_2.11-0.9.0.1 drwxr-xr-x 7 root root 94 Oct 5 03:23 kafka_2.11-0.9.0.1 -rw-r--r-- 1 root root 35650542 Oct 5 03:02 kafka_2.11-0.9.0.1.tgz lrwxrwxrwx 1 root root 21 Oct 5 02:44 zk02 -> /root/zookeeper-3.4.6 drwxr-xr-x 10 1000 1000 4096 Feb 20 2014 zookeeper-3.4.6 -rw-r--r-- 1 root root 17699306 Oct 5 02:43 zookeeper-3.4.6.tar.gz [[email protected] ~]#
linux-node3
[[email protected] ~]# ll total 202032 -rw-------. 1 root root 1011 Aug 26 18:00 anaconda-ks.cfg lrwxrwxrwx 1 root root 17 Oct 5 02:41 jdk -> /root/jdk1.7.0_79 drwxr-xr-x 8 10 143 4096 Apr 11 2015 jdk1.7.0_79 -rw-r--r-- 1 root root 153512879 Oct 5 02:39 jdk-7u79-linux-x64.tar.gz lrwxrwxrwx 1 root root 24 Oct 5 03:04 kafka03 -> /root/kafka_2.11-0.9.0.1 drwxr-xr-x 7 root root 94 Oct 5 03:23 kafka_2.11-0.9.0.1 -rw-r--r-- 1 root root 35650542 Oct 5 03:03 kafka_2.11-0.9.0.1.tgz lrwxrwxrwx 1 root root 21 Oct 5 02:44 zk03 -> /root/zookeeper-3.4.6 drwxr-xr-x 10 1000 1000 4096 Feb 20 2014 zookeeper-3.4.6 -rw-r--r-- 1 root root 17699306 Oct 5 02:43 zookeeper-3.4.6.tar.gz [[email protected] ~]#
kafka文件查看(每个文件的broker.id 不一样 和zk的myid差不多)
linux-node1
[[email protected] config]# cat server.properties broker.id=1 port=9092 listeners=PLAINTEXT://:9092 num.network.threads=8 num.io.threads=10 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/opt/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 zookeeper.connection.timeout.ms=6000 message.max.byte=5242880 default.replication.factor=2 replica.fetch.max.bytes=5242880 [[email protected] config]# pwd /root/kafka01/config [[email protected] config]#
linux-node2
[[email protected] config]# cat server.properties broker.id=2 port=9092 listeners=PLAINTEXT://:9092 num.network.threads=8 num.io.threads=10 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/opt/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 zookeeper.connection.timeout.ms=6000 message.max.byte=5242880 default.replication.factor=2 replica.fetch.max.bytes=5242880 [[email protected] config]#
linux-node3
[[email protected] kafka03]# cat config/server.properties broker.id=3 port=9092 listeners=PLAINTEXT://:9092 num.network.threads=8 num.io.threads=10 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/opt/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=192.168.56.11:2181,192.168.56.12:2181,192.168.56.13:2181 zookeeper.connection.timeout.ms=6000 message.max.byte=5242880 default.replication.factor=2 replica.fetch.max.bytes=5242880 [[email protected] kafka03]#
分别启动kafka
/root/kafka01/config ./kafka-server-start.sh -daemon ../config/server.properties
测试
./kafka-topics.sh --zookeeper localhost:2181 --create --topic ttt --partitions 2 --replication-factor 2 ./kafka-topics.sh --zookeeper 192.168.56.12:2181 --list ./kafka-topics.sh --zookeeper 192.168.56.12:2181 --describe --topic ttt
本文出自 “砖家博客” 博客，请务必保留此出处http://wsxxsl.blog.51cto.com/9085838/1970389
以上是关于kafka集群的部署的主要内容，如果未能解决你的问题，请参考以下文章