zookeeper集群起不起来

Posted 2023-04-24

参考技术A （1）解压为zookeepertar -xf -C /home/myuser/zookeeper/
复制zookeeper文件夹3份，分别重名名为zookeeperA，zookeeperB，zookeeperC。 并且创建数据快照以及日志存放文件夹，命名为zooA，zooB，zooC。 （2）编辑对应的zookeeper配置文件，复制zookeeperconf下zoo_sample.cfg为zoo.cfgcd /home/myuser/zookeeperA/conf
cp zoo_sample.cfg zoo.cfg
（3）修改zoo.cfg# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/home/myuser/zooA/data
# the port at which the clients will connect
clientPort=2181
# ZooKeeper server and its port no. # ZooKeeper ensemble should know about every other machine in the ensemble # specify server id by creating 'myid' file in the dataDir # use hostname instead of IP address for convenient maintenance
server.1=127.0.0.1:2888:3888
server.2=127.0.0.1:2988:3988
server.3=127.0.0.1:2088:3088
#
# Be sure to read the maintenance section of the
# administrator guide before turning>tickTime：心跳时间，为了确保连接存在的，以毫秒为单位，最小超时时间为两个心跳时间
initLimit：多少个心跳时间内，允许其他server连接并初始化数据，如果ZooKeeper管理的数据较大，则应相应增大这个值
clientPort：服务的监听端口
dataDir：用于存放内存数据库快照的文件夹，同时用于集群的myid文件也存在这个文件夹里（注意：一个配置文件只能包含一个dataDir字样，即使它被注释掉了。）
dataLogDir：用于单独设置transaction log的目录，transaction log分离可以避免和普通log还有快照的竞争
syncLimit：多少个tickTime内，允许follower同步，如果follower落后太多，则会被丢弃。
（4）创建myid文件
cd /home/myuser/zooA/data
sudo sh -c 'echo "1" >> myid'
其他文件夹类似创建myid文件，zookeeperB为2，zookeeperC为3
（5）启动zookeeper
cd /home/myuser/zookeeperA/bin
sudo sh zkServer.sh start
查看zookeeper状态[root@weibo bin]# sh zkServer.sh status
JMX enabled by default
Using config: /home/weibo/zookeeperA/bin/../conf/zoo.cfg
Mode: follower
启动OK，依次启动另外两台zookeeper，启动第一台zookeeper后，你可以观察bin下的zookeeper.out可以看到报错，connection
refused，没有关系，zookeeper需要等待其他另个节点的加入，全部启动之后就正常了。
（6）客户端连接zookeeper
[root@weibo bin]# sh zkCli.sh
Connecting to localhost:2181
2013-05-10 15:00:25,363 [myid:] - INFO [main:Environment@100] - Client environment:zookeeper.version=3.4.5-1392090, built>configs：保存上传的配置文件信息
clusterstate.json：集群状态json
aliases：别名json
live_node：当solr服务器启动的时候，会注册到这里
overseer：保存shard信息
overseer_elect：节点选举
collections：所有的collection

防火墙导致zookeeper集群异常，kafka起不来

问题描述：

主机信息：

IP	hostname
10.0.0.10	host10
10.0.0.12	host12
10.0.0.13	host13

在这三台主机上部署一套zookeeper&kafka集群环境的时候，zk集群进程和端口都起来了。然后在启动kafka的时候，报错了，提示连不上zk。
因为该环境要求必须开启防火墙，所以想到应该是因为2181端口没有开放，所以kafka去连zk，提示连不上。于是在修改了防火墙，打开了2181端口和kafka的9092端口：

[[email protected] ~]#  firewall-cmd --list-ports
2181/tcp  9092/tcp 
[[email protected] ~]#

然后重新启动kafka，依旧报错，报错信息如下：

[[email protected] ~]# /usr/local/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh /usr/local/kafka_2.11-0.11.0.0/config/server.properties
[2018-06-15 11:33:25,244] INFO KafkaConfig values:
        advertised.host.name = null
......   省略若干信息
[2018-06-15 11:33:26,693] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:28,219] INFO Opening socket connection to server 10.0.0.12/10.0.0.12181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:28,221] INFO Socket connection established to 10.0.0.12/10.0.0.12181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:28,222] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:28,910] INFO Opening socket connection to server 10.0.0.13/10.0.0.13181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:28,911] INFO Socket connection established to 10.0.0.13/10.0.0.13181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:28,913] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:29,901] INFO Opening socket connection to server 10.0.0.10/10.0.0.10181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:29,902] INFO Socket connection established to 10.0.0.10/10.0.0.10181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:29,905] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:31,133] INFO Opening socket connection to server 10.0.0.12/10.0.0.12181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:31,135] INFO Socket connection established to 10.0.0.12/10.0.0.12181, initiating session (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:31,136] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:31,379] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2018-06-15 11:33:31,553] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2018-06-15 11:33:31,555] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server ‘10.0.0.10:2181,10.0.0.12:2181,10.0.0.13:2181‘ with timeout of 6000 ms
        at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
        at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
        at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:103)
        at kafka.utils.ZkUtils$.apply(ZkUtils.scala:85)
        at kafka.server.KafkaServer.initZk(KafkaServer.scala:338)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:191)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
        at kafka.Kafka$.main(Kafka.scala:65)
        at kafka.Kafka.main(Kafka.scala)
[2018-06-15 11:33:31,555] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2018-06-15 11:33:31,562] INFO shutting down (kafka.server.KafkaServer)
[2018-06-15 11:33:31,568] INFO shut down completed (kafka.server.KafkaServer)
[2018-06-15 11:33:31,568] FATAL Exiting Kafka. (kafka.server.KafkaServerStartable)
[2018-06-15 11:33:31,571] INFO shutting down (kafka.server.KafkaServer)

查看kafka的配置，并未发现任何异常。于是检查zk自身是否OK。但是进程、端口都一切正常。
使用./zkCli.sh去检查zk是否正常启动，出现了报错：

[[email protected] bin]# ./zkCli.sh
Connecting to localhost:2181
2018-06-15 14:44:05,215 [myid:] - INFO  [main:[email protected]] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
......    #省略若干信息
2018-06-15 14:44:05,225 [myid:] - INFO  [main:[email protected]] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 [email protected]86f2f1
Welcome to ZooKeeper!
2018-06-15 14:44:05,255 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-06-15 14:44:05,334 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2018-06-15 14:44:05,348 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
[zk: localhost:2181(CONNECTING) 0] 2018-06-15 14:44:05,564 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-06-15 14:44:05,565 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-06-15 14:44:05,573 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-06-15 14:44:07,626 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-06-15 14:44:07,627 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2018-06-15 14:44:07,629 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-06-15 14:44:07,929 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-06-15 14:44:07,930 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Socket connection established to localhost/127.0.0.1:2181, initiating session
2018-06-15 14:44:07,932 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-06-15 14:44:09,329 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
2018-06-15 14:44:09,330 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
2018-06-15 14:44:09,332 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2018-06-15 14:44:09,464 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)

分析：

查看了网上很多的文章，有以下几种说法：
1、zk的myid文件和配置文件server.xx的编号对不上：
2、zk只启动了一个节点，其他节点没有起来

排查：
1、检查myid编号和配置文件，是OK的，能对上：

[[email protected] ~]# cat /data/zookeeper/myid
1
[[email protected] ~]#
[[email protected] ~]# vim /usr/local/zookeeper-3.4.10/conf/zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
......  #省略若干
server.1=10.0.0.10:2888:3888   #server.1和myid文件的1是能对上的
server.2=10.0.0.12:2888:3888
server.3=10.0.0.13:2888:3888

2、zookeeper的配置文件zoo.cfg里面配置了三个server，实际上其他两个节点没有起来，因此根据zookeeper的选举算法，当整个集群超过半数机器宕机，zookeeper会认为集群处于不可用状态。

检查了三个节点的服务，都是起来了的。
试着将其中一台zk节点10.0.0.10，修改zoo.cfg，注释掉配置的其他两个server，只留本机的server配置：

[[email protected] ~]# vim /usr/local/zookeeper-3.4.10/conf/zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
......  #省略若干
server.1=10.0.0.10:2888:3888   #注释掉server.2和server.3，只剩下本机节点
#server.2=10.0.0.12:2888:3888
#server.3=10.0.0.13:2888:3888

然后使用zkCli.sh去检测zk状态，就是OK的：

[[email protected] ~]# cd /usr/local/zookeeper-3.4.10/bin/
[[email protected] bin]# ./zkCli.sh
Connecting to localhost:2181
2018-06-15 15:58:10,586 [myid:] - INFO  [main:[email protected]] - Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
......  #省略若干
2018-06-15 15:58:10,597 [myid:] - INFO  [main:[email protected]] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 [email protected]
Welcome to ZooKeeper!
2018-06-15 15:58:10,627 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2018-06-15 15:58:10,708 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session
[zk: localhost:2181(CONNECTING) 0] 2018-06-15 15:58:10,733 [myid:] - INFO  [main-SendThread(localhost:2181):[email protected]] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x164023cac190003, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: localhost:2181(CONNECTED) 0]

于是想到是防火墙只开了zookeeper的2181端口。zk有三个端口：
3个端口的作用
1、2181：对cline端提供服务
2、3888：选举leader使用
3、2888：集群内机器通讯使用（Leader监听此端口）

因为防火墙的作用，三个节点之间2888和3888端口都是互相不通的，因此集群之前没有办法通信，可能就会以为是剩下两个节点都挂掉了。于是就出现了上面的报错

解决：

1、重新修改防火墙，打开2888和3888端口

[[email protected] ~]#  firewall-cmd --list-ports
 3888/tcp 2181/tcp 2888/tcp  9092/tcp 
[[email protected] ~]#

2、为保证万一，把整个zk集群全部按照顺序重启了一遍
3、三个几点依次都用zkCli.sh去连了一下，状态都是OK的
4、重新启动kafka，就OK啦