记一次Redisson连接异常引起的OOM

Posted 2022-02-01 第三眼的思绪

线上服务内存使用越来越高，日志出现较大量以下异常，内存始终无法释放

org.redisson.client.RedisTimeoutException: Command still hasn't been written into connection! Increase nettyThreads and/or retryInterval settings. Payload size in bytes: 0. Node sourc
e: NodeSource [slot=0, addr=null, redisClient=null, redirect=null, entry=MasterSlaveEntry [masterEntry=[freeSubscribeConnectionsAmount=0, freeSubscribeConnectionsCounter=value:1:queue
:0, freeConnectionsAmount=2, freeConnectionsCounter=value:5:queue:0, freezed=false, freezeReason=null, client=[addr=redis://172.16.0.x:9736], nodeType=MASTER, firstFail=0]]], connecti
on: RedisConnection@908715113 [redisClient=[addr=redis://172.16.0.x:9736], channel=[id: 0x1a19209d, L:/172.16.0.x:34420 - R:172.16.0.6/172.16.0.x:9736], currentCommand=CommandData [pr
omise=RedissonPromise [promise=ImmediateEventExecutor$ImmediatePromise@38ee0cdc(failure: java.util.concurrent.CancellationException)], command=(EVAL), params=[while true do local firs
tThreadId2 = redis.call('lindex', KEYS[2], 0);if firstThreadId2 == false the..., 4, redis:distributed:lock:coupon:order:refresh_pay_result, redisson_lock_queue:redis:distributed:lock
:coupon:order:refresh_pay_result, redisson_lock_timeout:redis:distributed:lock:coupon:order:refresh_pay_result, redisson_lock__channel:redis:distributed:lock:coupon:order:refresh_
pay_result, 0, 30000, 17d3e90c-449d-4cb7-aa38-faf3d87b5bc4:169, 1629769615010], codec=org.redisson.client.codec.LongCodec]], command: (EVAL), params: [while true do local firstThread
Id2 = redis.call('lindex', KEYS[2], 0);if firstThreadId2 == false the..., 4, redis:distributed:lock:coupon:status:lock:101305217069507712, redisson_lock_queue:redis:distributed:lock:
coupon:status:lock:101305217069507712, redisson_lock_timeout:redis:distributed:lock:coupon:status:lock:101305217069507712, redisson_lock__channel:redis:distributed:lock:coupon:sta
tus:lock:101305217069507712, 0, 30000, 17d3e90c-449d-4cb7-aa38-faf3d87b5bc4:127, 1631374416987] after 3 retry attempts
        at org.redisson.command.RedisExecutor$2.run(RedisExecutor.java:200) ~[redisson-3.12.4.jar!/:3.12.4]
        at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:672) ~[netty-all-4.1.58.Final.jar!/:4.1.58.Final]
        at io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:747) ~[netty-all-4.1.58.Final.jar!/:4.1.58.Final]
        at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:472) ~[netty-all-4.1.58.Final.jar!/:4.1.58.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.58.Final.jar!/:4.1.58.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

找到RedisExecutor类200行左右，异常后执行

attemptPromise.tryFailure(exception);

因org.redisson.client.handler.CommandsQueue中使用异步操作，大量对象存储在无界队列中无法释放。