zookeeper 无法启动 ERROR org.apache.zookeeper.server.quorum.QuorumPeer: Unable to load database on disk

Posted 2020-12-14 bioamin

环境：cdh集群5.15 ，redhat7

有一台zookeeper无法启动，查看日志报错如下

2020-07-04 14:24:23,880 ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial.
2020-07-04 14:24:23,881 ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial.
2020-07-04 14:24:23,881 ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial.
2020-07-04 14:24:23,881 ERROR org.apache.zookeeper.server.persistence.Util: Last transaction was partial.
2020-07-04 14:24:23,882 ERROR org.apache.zookeeper.server.quorum.QuorumPeer: Unable to load database on disk
java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
        at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:610)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:629)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:595)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:677)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:209)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:530)
        at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:520)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:167)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)
2020-07-04 14:24:23,891 ERROR org.apache.zookeeper.server.quorum.QuorumPeerMain: Unexpected exception, exiting abnormally
java.lang.RuntimeException: Unable to run quorum server
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:578)
        at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:520)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:167)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:79)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readInt(DataInputStream.java:392)
        at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
        at org.apache.zookeeper.server.persistence.FileHeader.deserialize(FileHeader.java:64)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.inStreamCreated(FileTxnLog.java:610)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.createInputArchive(FileTxnLog.java:629)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.goToNextLog(FileTxnLog.java:595)
        at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:677)
        at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:209)
        at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
        at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:530)

 

报错解析：  磁盘写满导致无法写入数据导致

 

问题解决：进入到zookeeper的数据目录，删除文件大小为0的日志文件，然后重启节点，重启成功

[dip@dip007 zookeeper]$ cd version-2/
[dip@dip007 version-2]$ ls
acceptedEpoch  log.210069890e  log.21006b70bc  log.21006d6e04  log.21006f2bbc  log.21006f3149  snapshot.21006a5af3  snapshot.21006c445a  snapshot.21006ee9d6
currentEpoch   log.21006a5af4  log.21006c445c  log.21006ee9d8  log.21006f2d34  log.21006f317f  snapshot.21006b70ba  snapshot.21006d6e03
[dip@dip007 version-2]$ ll
总用量 342408
-rw-r--r-- 1 zookeeper zookeeper        2 4月  11 10:35 acceptedEpoch
-rw-r--r-- 1 zookeeper zookeeper        2 5月   7 22:02 currentEpoch
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   6 15:29 log.210069890e
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   6 21:43 log.21006a5af4
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   7 01:07 log.21006b70bc
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   7 02:41 log.21006c445c
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   7 15:48 log.21006d6e04
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   7 21:23 log.21006ee9d8
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   7 21:24 log.21006f2bbc
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   7 21:32 log.21006f2d34
-rw-r--r-- 1 zookeeper zookeeper 67108880 5月   7 22:01 log.21006f3149
-rw-r--r-- 1 zookeeper zookeeper        0 5月   7 22:02 log.21006f317f
-rw-r--r-- 1 zookeeper zookeeper 33998220 5月   6 15:29 snapshot.21006a5af3
-rw-r--r-- 1 zookeeper zookeeper 35208598 5月   6 21:43 snapshot.21006b70ba
-rw-r--r-- 1 zookeeper zookeeper 35673050 5月   7 01:07 snapshot.21006c445a
-rw-r--r-- 1 zookeeper zookeeper 35637204 5月   7 02:41 snapshot.21006d6e03
-rw-r--r-- 1 zookeeper zookeeper 33990250 5月   7 15:48 snapshot.21006ee9d6

[root@dip007 version-2]# rm log.21006f317f