MongoDB的mongos实例因无法分配mlock内存挂掉

Posted 2020-10-14

问题版本

mongodb-v3.4.4

问题描述

mongos两天死了两次，死前遗言只有日志：

2017-11-01T11:25:27.135+0800 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to mlock: Cannot allocate memory
2017-11-01T11:25:27.140+0800 I -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246
2017-11-01T11:25:27.140+0800 I -        [NetworkInterfaceASIO-TaskExecutorPool-1-0]

***aborting after fassert() failure

2017-11-01T11:25:27.182+0800 I NETWORK  [thread2] connection accepted from 100.97.227.0:37132 #282914 (1156 connections now open)
2017-11-01T11:25:27.191+0800 F -        [NetworkInterfaceASIO-TaskExecutorPool-1-0] Got signal: 6 (Aborted).

 0x55db5fab1241 0x55db5fab0339 0x55db5fab081d 0x7f848d578390 0x7f848d1d3428 0x7f848d1d502a 0x55db5f2e90e7 0x55db5f3452cd 0x55db5f407733 0x55db5f4039ce 0x55db5f40538c 0x55db5f3da9c7 0x55db5f3fc16f 0x55db5f3fedaf 0x55db5f7dba02 0x55db5f7dc1be 0x55db5f7ddf64 0x55db5f7a69d4 0x55db5f7b9498 0x55db5f7b9ae2 0x55db5fb8edd4 0x55db5fb8efd1 0x55db5f7cf91f 0x55db5fe8e120 0x7f848d56e6ba 0x7f848d2a482d

关键点：Failed to mlock: Cannot allocate memory和Got signal: 6 (Aborted).

排查

mongodb的jira上有两个issue跟这个一模一样：

1. SERVER-29086
2. SERVER-28997

分析

如SERVER-28997

SaslSCRAMSHA1ClientConversations have a SCRAMSecrets which they ‘ll pull out of the cache. SCRAMSecrets allocate secure storage in their default constructor, so they may be populated. Instead, SaslSCRAMSHA1ClientConversation and the cache should store shared_ptrs to SCRAMSecret.

大致意思就是说SaslSCRAMSHA1ClientConversation这玩意儿每次初始化对象都会申请一片安全的存储，这个会填充memlock区域，正确的做法应该是共享这片区域。

处理

按照SERVER-29086的意思，可以调整memlock大小，来缓解这个问题。

按照SERVER-28997的意思，这是3.4.4版本的bug，所以可以升级到bug-fixed版本3.4.6，这是最终的解决方法。

注意：要多多关注各种issue，jira，邮件列表。