OOM 手记

Posted 2021-09-10 agentd

最近项目经常会被 OOM kill 掉，但是没有在 /var/log/messages 看到 OOM 日志。
其实不同 Linux 发行版 OOM 日志在不同的地方，但是可以通过 dmesg 查看。

[33350932.058517] Task in /kubepods/burstable/podbed004f6-87f6-415f-8b42-696e12d6096a/851f02a6960c8b872559de9e29feda1b8ed41d397096d95d3dc8d6f74e9de061 killed as a result of limit of /kubepods/burstable/podbed004f6-87f6-415f-8b42-696e12d6096a
[33350932.058524] memory: usage 14680064kB, limit 14680064kB, failcnt 811012
[33350932.058525] memory+swap: usage 14680064kB, limit 9007199254740988kB, failcnt 0
[33350932.058526] kmem: usage 194960kB, limit 9007199254740988kB, failcnt 0
[33350932.058526] Memory cgroup stats for /kubepods/burstable/podbed004f6-87f6-415f-8b42-696e12d6096a: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:0KB inactive_file:0KB active_file:0KB unevictable:0KB
[33350932.058540] Memory cgroup stats for /kubepods/burstable/podbed004f6-87f6-415f-8b42-696e12d6096a/3bf687dcb3254c8e7b66ad7b5a93918f65076cae647d84424f6f590ef13ba451: cache:0KB rss:0KB rss_huge:0KB shmem:0KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:48KB inactive_file:0KB active_file:0KB unevictable:0KB
[33350932.058550] Memory cgroup stats for /kubepods/burstable/podbed004f6-87f6-415f-8b42-696e12d6096a/851f02a6960c8b872559de9e29feda1b8ed41d397096d95d3dc8d6f74e9de061: cache:0KB rss:14482160KB rss_huge:0KB shmem:0KB mapped_file:528KB dirty:0KB writeback:3960KB swap:0KB inactive_anon:0KB active_anon:14485012KB inactive_file:16KB active_file:8KB unevictable:0KB
[33350932.058559] Tasks state (memory values in pages):
[33350932.058560] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[33350932.059085] [  53131]     0 53131      256        1    32768        0          -998 pause
[33350932.059090] [  53484]     0 53484     3804      826    73728        0           945 sh
[33350932.059093] [  53511]     0 53511    27556      708   253952        0           945 sshd
[33350932.059096] [  53565]     0 53565    21333     1104   212992        0           945 su
[33350932.059099] [  53567]     0 53567    27538      711    86016        0           945 opagent
[33350932.059101] [  53579]  1602 53579 12617494  3622901 34770944        0           945 java
[33350932.059104] [  59796]     0 59796     1933      180    57344        0           945 sleep
[33350932.059248] Memory cgroup out of memory: Kill process 53579 (java) score 1934 or sacrifice child
[33350932.065172] Killed process 53579 (java) total-vm:50469976kB, anon-rss:14454940kB, file-rss:36664kB, shmem-rss:0kB
[33350933.005245] oom_reaper: reaped process 53579 (java), now anon-rss:0kB, file-rss:8kB, shmem-rss:0kB

Docker 给了 15G 的内存，JVM 分配了 12G，还有各种 Buffer 和 Cache 的内存，结果就超过了限制，被 OOM 了。

同事听到 OOM kill，第一反应大部分是难道 JVM 内存泄漏了？

其实 OOM kill 是 Linux 服务器本身的内存满了，不是 JVM 的堆内存满了。

参考链接
https://stackoverflow.com/que...