安装phoenix时初始化连接hbase异常导致hbase节点挂掉

Posted 2021-06-05 胖子学习天地

 

重启Hbase后，看到hbase的几个节点陆续挂掉，然后发现有Regions in transition状态。

 

在命令模式下使用Hbase 修复命令：

bin/hbase hbck -repair

提示错误信息，显示存在某个lock文件。

java.util.concurrent.ExecutionException: org.apache.hadoop.fs.FileAlreadyExistsException: /hbase/.tmp/hbase-hbck.lock for client 192.168.80.200 already exists
    at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.startFile(FSDirWriteFileOp.java:388)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2478)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2375)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:791)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:469)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1000)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2916)

    at java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.util.concurrent.FutureTask.get(FutureTask.java:206)
    at org.apache.hadoop.hbase.util.HBaseFsck.checkAndMarkRunningHbck(HBaseFsck.java:495)
    at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:547)
    at org.apache.hadoop.hbase.util.HBaseFsck.exec(HBaseFsck.java:3799)
    at org.apache.hadoop.hbase.util.HBaseFsck$HBaseFsckTool.run(HBaseFsck.java:3623)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
    at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3611)
Caused by: org.apache.hadoop.fs.FileAlreadyExistsException: /hbase/.tmp/hbase-hbck.lock for client 192.168.80.200 already exists

在hdfs上找到该文件删除，然后继续使用修复命令bin/hbase hbck -repair ，此时，看到修复成功。

 

 

hbase/.tmp/hbase-hbck.lock