解决levelDB导致ActiveMQ一直占用1个CPU的100%使用率的问题

Posted 2021-05-26 轻轻的烟雾

最开始的问题表现为，ActiveMQ一直占用1个CPU的100%使用率。
一直也没找到问题在哪里。最后去看日志，发现日志更新特快，内容如下。
然后我想，之所以100%的CPU，也肯定是因为这个写日志的原因。

2021-05-25 22:28:29,606 | WARN | Could not load message seq: 1483 from DataLocator(39b286d3a3, 1101) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,606 | WARN | No reader available for position: 39b286d852, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,607 | WARN | Could not load message seq: 1484 from DataLocator(39b286d852, 2100) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,607 | WARN | No reader available for position: 39b286e0e8, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,608 | WARN | Could not load message seq: 1485 from DataLocator(39b286e0e8, 1101) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,609 | WARN | No reader available for position: 39b286e597, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,609 | WARN | Could not load message seq: 1486 from DataLocator(39b286e597, 2100) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,610 | WARN | No reader available for position: 39b286ee2d, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,611 | WARN | Could not load message seq: 1487 from DataLocator(39b286ee2d, 1101) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,611 | WARN | No reader available for position: 39b286f2dc, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,612 | WARN | Could not load message seq: 1488 from DataLocator(39b286f2dc, 2100) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,613 | WARN | No reader available for position: 39b286fb72, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,613 | WARN | Could not load message seq: 1489 from DataLocator(39b286fb72, 1101) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,614 | WARN | No reader available for position: 39b2870021, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,614 | WARN | Could not load message seq: 1490 from DataLocator(39b2870021, 2100) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,615 | WARN | No reader available for position: 39b28708b7, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,616 | WARN | Could not load message seq: 1491 from DataLocator(39b28708b7, 1101) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,616 | WARN | No reader available for position: 39b2870d66, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,617 | WARN | Could not load message seq: 1492 from DataLocator(39b2870d66, 2101) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,618 | WARN | No reader available for position: 39b28715fd, log_infos: {324641837082=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/0000004b9629501a.log,324641837082,104859415), 518839757265=LogInfo(/usr/local/apache-activemq-5.11.1/data/leveldb/00000078cd4269d1.log,518839757265,0)} | org.apache.activemq.leveldb.RecordLog | ActiveMQ Broker[localhost] Scheduler
2021-05-25 22:28:29,618 | WARN | Could not load message seq: 1493 from DataLocator(39b28715fd, 1101) | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ Broker[localhost] Scheduler
20

View Code

 

然后通过日志，搜索，网上有几个帖子，但是没有人回复。

无奈之下，尝试给报错的日志文件（就是日志里0000004b9629501a.log、00000078cd4269d1.log），删除掉（我是采取mv到上级目录的办法）。然后重启服务。

然后，仍然报错。不过果然没有100%CPU了，错误日志没有持续快速记录了，但是会定时出现：

2021-05-26 00:04:26,027 | INFO | Stopping BrokerService[localhost] due to exception, java.io.EOFException: File \'/usr/local/apache-activemq-5.11.1/data/leveldb/0000000000000000.log\' offset: 43428088266 | org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException handler.
java.io.EOFException: File \'/usr/local/apache-activemq-5.11.1/data/leveldb/0000000000000000.log\' offset: 43428088266
at org.apache.activemq.leveldb.RecordLog$LogReader.read(RecordLog.scala:389)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(RecordLog.scala:654)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.RecordLog$$anonfun$read$2.apply(RecordLog.scala:654)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.RecordLog.get_reader(RecordLog.scala:644)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.RecordLog.read(RecordLog.scala:654)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.getMessage(LevelDBClient.scala:1335)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.apply(LevelDBClient.scala:1274)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$queueCursor$1.apply(LevelDBClient.scala:1271)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1359)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1$$anonfun$apply$mcV$sp$12.apply(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$RichDB.check$4(LevelDBClient.scala:323)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$RichDB.cursorRange(LevelDBClient.scala:325)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply$mcV$sp(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$collectionCursor$1.apply(LevelDBClient.scala:1358)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.usingIndex(LevelDBClient.scala:1038)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient$$anonfun$might_fail_using_index$1.apply(LevelDBClient.scala:1044)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:549)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBClient.scala:1357)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.scala:1271)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:735)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverNextMessages(LevelDBStore.scala:860)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:109)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:381)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:142)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:159)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1897)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2119)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1116)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:909)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:100)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:144)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.11.1.jar:5.11.1]
at java.util.TimerThread.mainLoop(Timer.java:555)[:1.8.0]
at java.util.TimerThread.run(Timer.java:505)[:1.8.0]
2021-05-26 00:04:26,031 | INFO | Apache ActiveMQ 5.11.1 (localhost, ID:file-46242-1621958574889-0:3) is shutting down | org.apache.activemq.broker.BrokerService | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,055 | INFO | Stopped LevelDB[/usr/local/apache-activemq-5.11.1/data/leveldb] | org.apache.activemq.leveldb.LevelDBStore | LevelDB IOException handler.
2021-05-26 00:04:26,060 | INFO | Connector openwire stopped | org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,063 | INFO | Connector amqp stopped | org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,065 | INFO | Connector stomp stopped | org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,067 | INFO | Connector mqtt stopped | org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,127 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@5b2a0d2a:ActiveMQ.DLQ,batchResetNeeded=false,size=2383,cacheEnabled=false,maxBatchSize:200,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=LevelDBMessageStore(queue://ActiveMQ.DLQ,3) - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
at org.apache.activemq.leveldb.LevelDBClient.handleFailure$1(LevelDBClient.scala:546)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:548)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:1044)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBClient.scala:1357)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.scala:1271)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:735)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverNextMessages(LevelDBStore.scala:860)[activemq-leveldb-store-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:109)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:381)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:142)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:159)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1897)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2119)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1116)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:909)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:100)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:144)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.11.1.jar:5.11.1]
at java.util.TimerThread.mainLoop(Timer.java:555)[:1.8.0]
at java.util.TimerThread.run(Timer.java:505)[:1.8.0]
2021-05-26 00:04:26,130 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@5b2a0d2a:ActiveMQ.DLQ,batchResetNeeded=false,size=2383,cacheEnabled=false,maxBatchSize:200,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=LevelDBMessageStore(queue://ActiveMQ.DLQ,3) - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ Broker[localhost] Scheduler
java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:384)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:142)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:159)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1897)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2119)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1116)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:909)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:100)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:144)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.11.1.jar:5.11.1]
at java.util.TimerThread.mainLoop(Timer.java:555)[:1.8.0]
at java.util.TimerThread.run(Timer.java:505)[:1.8.0]
2021-05-26 00:04:26,132 | INFO | Connector ws stopped | org.apache.activemq.broker.TransportConnector | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,132 | ERROR | Problem retrieving message for browse | org.apache.activemq.broker.region.Queue | ActiveMQ Broker[localhost] Scheduler
java.lang.RuntimeException: java.lang.RuntimeException: org.apache.activemq.broker.SuppressReplyException: org.apache.activemq.broker.SuppressReplyException: ShutdownBrokerInitiated
at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:145)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:159)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1897)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2119)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1116)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:909)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue.access$100(Queue.java:100)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.broker.region.Queue$2.run(Queue.java:144)[activemq-broker-5.11.1.jar:5.11.1]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)[activemq-client-5.11.1.jar:5.11.1]
at java.util.TimerThread.mainLoop(Timer.java:555)[:1.8.0]
at java.util.TimerThread.run(Timer.java:505)[:1.8.0]
2021-05-26 00:04:26,142 | INFO | Apache ActiveMQ 5.11.1 (localhost, ID:file-46242-1621958574889-0:3) uptime 30.232 seconds | org.apache.activemq.broker.BrokerService | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,144 | INFO | Apache ActiveMQ 5.11.1 (localhost, ID:file-46242-1621958574889-0:3) is shutdown | org.apache.activemq.broker.BrokerService | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,146 | INFO | Closing org.apache.activemq.xbean.XBeanBrokerFactory$1@11de56e6: startup date [Wed May 26 00:03:55 CST 2021]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,164 | INFO | Destroying Spring FrameworkServlet \'dispatcher\' | /admin | IOExceptionHandler: stopping BrokerService[localhost]
2021-05-26 00:04:26,229 | INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@7f5eae0f: startup date [Wed May 26 00:04:26 CST 2021]; root of context hierarchy | org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2021-05-26 00:04:26,312 | INFO | Using Persistence Adapter: LevelDB[/usr/local/apache-activemq-5.11.1/data/leveldb] | org.apache.activemq.broker.BrokerService | main
2021-05-26 00:04:26,315 | INFO | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | main
2021-05-26 00:04:26,349 | INFO | Apache ActiveMQ 5.11.1 (localhost, ID:file-46242-1621958574889-0:4) is starting | org.apache.activemq.broker.BrokerService | main
2021-05-26 00:04:26,357 | INFO | Usage(default:memory:queue://ActiveMQ.DLQ:memory) limit=1048576000 should be smaller than its parent limit=668309914 | org.apache.activemq.usage.Usage | main
2021-05-26 00:04:26,366 | INFO | Listening for connections at: tcp://file:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2021-05-26 00:04:26,370 | INFO | Connector openwire started | org.apache.activemq.broker.TransportConnector | main
2021-05-26 00:04:26,373 | INFO | Listening for connections at: amqp://file:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2021-05-26 00:04:26,376 | INFO | Connector amqp started | org.apache.activemq.broker.TransportConnector | main
2021-05-26 00:04:26,379 | INFO | Listening for connections at: stomp://file:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2021-05-26 00:04:26,391 | INFO | Connector stomp started | org.apache.activemq.broker.TransportConnector | main
2021-05-26 00:04:26,397 | INFO | Listening for connections at: mqtt://file:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2021-05-26 00:04:26,403 | INFO | Connector mqtt started | org.apache.activemq.broker.TransportConnector | main
2021-05-26 00:04:26,409 | INFO | Listening for connections at ws://file:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.ws.WSTransportServer | main
2021-05-26 00:04:26,410 | INFO | Connector ws started | org.apache.activemq.broker.TransportConnector | main
2021-05-26 00:04:26,412 | INFO | Apache ActiveMQ 5.11.1 (localhost, ID:file-46242-1621958574889-0:4) started | org.apache.activemq.broker.BrokerService | main
2021-05-26 00:04:26,418 | INFO | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2021-05-26 00:04:26,421 | WARN | Store limit is 102400 mb (current store usage is 2400 mb). The data directory: /usr/local/apache-activemq-5.11.1/data/leveldb only has 9208 mb of usable space - resetting to maximum available disk space: 11608 mb | org.apache.activemq.broker.BrokerService | main
2021-05-26 00:04:26,424 | WARN | Temporary Store limit is 51200 mb, whilst the temporary data directory: /usr/local/apache-activemq-5.11.1/data/localhost/tmp_storage only has 9208 mb of usable space - resetting to maximum available 9208 mb. | org.apache.activemq.broker.BrokerService | main
2021-05-26 00:04:26,583 | INFO | ActiveMQ WebConsole available at http://0.0.0.0:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2021-05-26 00:04:26,587 | INFO | Initializing Spring FrameworkServlet \'dispatcher\' | /admin | main
2021-05-26 00:04:26,627 | INFO | jolokia-agent: No access restrictor found at classpath:/jolokia-access.xml, access to all MBeans is allowed | /api | main

View Code

 

仔细看了一下，发现一个存储空间配置过大：

Temporary Store limit is 51200 mb

然后修改一下配置文件：

activemq.xml里面的 systemUsage 字段里面的配置。
然后解决这个问题。但是，还是上面的leveldb的IO问题仍然在。

然后，继续搜了一下 ActiveMQ 的消息持久化的方式，发现有很多，于是考虑换一种，不用 leveldb 了。
然后参考这里：https://www.cnblogs.com/binyue/p/5371479.html （感谢作者）
使用了 kahaDB。修改activemq.xml里面的persistenceAdapter：

<persistenceAdapter>
<kahaDB directory="${activemq.data}/activemq-data" journalMaxFileLength="16mb"/>
</persistenceAdapter>

directory : 指定持久化消息的存储目录

journalMaxFileLength : 指定保存消息的日志文件大小，具体根据你的实际应用配置

重启服务。问题解决。
只是，最后没搞明白为什么levelDB啥问题。

如有需要，联系 281099678