案例分析 - OOM的内存分析

Posted 2021-05-16 程序猿Ricky

当Linux系统由于内存不足而触发了oom killer时，会在内核log中输出相关的内存信息，按照信息类型分为这几部分来介绍。

• 第1部分

首先输出的是整机的内存信息，按照指标分行整理后显示如下：

Mem-Info:
active_anon:81208569
inactive_anon:209684 
isolated_anon:0#012 
active_file:72830 
inactive_file:121873 
isolated_file:0#012 
unevictable:15512 
dirty:276 writeback:0 
unstable:0#012 
slab_reclaimable:81306 
slab_unreclaimable:221415#012 
mapped:353261 
shmem:343535 
pagetables:168686 
bounce:0#012 
free:48511608 
free_pcp:48007 
free_cma:0

• 第2部分

第二部分输出的是各个zone中的数据：

Node 0 DMA free:15900kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 2639 63552 63552

Node 0 DMA32 free:253376kB min:1152kB low:1440kB high:1728kB active_anon:2410340kB inactive_anon:656kB active_file:1228kB inactive_file:540kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3569856kB managed:2702704kB mlocked:0kB dirty:12kB writeback:0kB mapped:1336kB shmem:676kB slab_reclaimable:616kB slab_unreclaimable:3592kB kernel_stack:288kB pagetables:8836kB unstable:0kB bounce:0kB free_pcp:8944kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 60912 60912

Node 0 Normal free:29219664kB min:26636kB low:33292kB high:39952kB active_anon:32259176kB inactive_anon:34304kB active_file:34076kB inactive_file:10832kB unevictable:808kB isolated(anon):0kB isolated(file):0kB present:63426048kB managed:62374864kB mlocked:808kB dirty:36kB writeback:0kB mapped:56396kB shmem:34540kB slab_reclaimable:44960kB slab_unreclaimable:124680kB kernel_stack:5376kB pagetables:62528kB unstable:0kB bounce:0kB free_pcp:20676kB local_pcp:404kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0

Node 1 Normal free:39051596kB min:28208kB low:35260kB high:42312kB active_anon:26142100kB inactive_anon:130872kB active_file:12536kB inactive_file:4488kB unevictable:5940kB isolated(anon):0kB isolated(file):0kB present:67108352kB managed:66057372kB mlocked:5940kB dirty:24kB writeback:0kB mapped:136580kB shmem:135844kB slab_reclaimable:27184kB slab_unreclaimable:81684kB kernel_stack:4912kB pagetables:57336kB unstable:0kB bounce:0kB free_pcp:12672kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0

Node 2 Normal free:28180kB min:28208kB low:35260kB high:42312kB active_anon:64495908kB inactive_anon:74632kB active_file:72kB inactive_file:72kB unevictable:300kB isolated(anon):0kB isolated(file):0kB present:67108352kB managed:66057372kB mlocked:300kB dirty:0kB writeback:0kB mapped:74628kB shmem:74928kB slab_reclaimable:42444kB slab_unreclaimable:116024kB kernel_stack:5056kB pagetables:126304kB unstable:0kB bounce:0kB free_pcp:10160kB local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:183 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0

Node 3 Normal free:28168kB min:28208kB low:35260kB high:42312kB active_anon:65309792kB inactive_anon:38812kB active_file:112kB inactive_file:100kB unevictable:608kB isolated(anon):0kB isolated(file):0kB present:67108352kB managed:66057372kB mlocked:608kB dirty:0kB writeback:0kB mapped:38908kB shmem:38944kB slab_reclaimable:33716kB slab_unreclaimable:118132kB kernel_stack:3904kB pagetables:134380kB unstable:0kB bounce:0kB free_pcp:10448kB local_pcp:24kB free_cma:0kB writeback_tmp:0kB pages_scanned:11087 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0

Node 4 Normal free:18663032kB min:28208kB low:35260kB high:42312kB active_anon:46300064kB inactive_anon:204960kB active_file:31340kB inactive_file:68368kB unevictable:2964kB isolated(anon):0kB isolated(file):0kB present:67108352kB managed:66057372kB mlocked:2964kB dirty:112kB writeback:0kB mapped:215784kB shmem:205364kB slab_reclaimable:36864kB slab_unreclaimable:101424kB kernel_stack:4064kB pagetables:99344kB unstable:0kB bounce:0kB free_pcp:33380kB local_pcp:440kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0

Node 5 Normal free:33982352kB min:28208kB low:35260kB high:42312kB active_anon:31160872kB inactive_anon:132740kB active_file:42032kB inactive_file:86300kB unevictable:2016kB isolated(anon):0kB isolated(file):0kB present:67108352kB managed:66057372kB mlocked:2016kB dirty:284kB writeback:0kB mapped:624336kB shmem:660584kB slab_reclaimable:33848kB slab_unreclaimable:103964kB kernel_stack:8112kB pagetables:64208kB unstable:0kB bounce:0kB free_pcp:35724kB local_pcp:488kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0

Node 6 Normal free:41998000kB min:28208kB low:35260kB high:42312kB active_anon:22811844kB inactive_anon:90712kB active_file:123296kB inactive_file:299740kB unevictable:45384kB isolated(anon):0kB isolated(file):0kB present:67108352kB managed:66057372kB mlocked:45384kB dirty:508kB writeback:0kB mapped:111280kB shmem:91940kB slab_reclaimable:70528kB slab_unreclaimable:134992kB kernel_stack:4256kB pagetables:49712kB unstable:0kB bounce:0kB free_pcp:34592kB local_pcp:588kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0

Node 7 Normal free:30806164kB min:28208kB low:35260kB high:42312kB active_anon:33944180kB inactive_anon:131048kB active_file:46628kB inactive_file:17052kB unevictable:4028kB isolated(anon):0kB isolated(file):0kB present:67108352kB managed:66056864kB mlocked:4028kB dirty:128kB writeback:0kB mapped:153796kB shmem:131320kB slab_reclaimable:35064kB slab_unreclaimable:101168kB kernel_stack:4032kB pagetables:72096kB unstable:0kB bounce:0kB free_pcp:25432kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0

特别提出的是，每个zone中的lowmem_reserve中的数据都是以page为单位的数值。它表示的是该zone中的内存被其他zone所申请走时需要考虑的预留内存。

• 第3部分
  buddy system相关信息：

Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB
Node 0 DMA32: 432*4kB (UEM) 394*8kB (UEM) 201*16kB (UEM) 113*32kB (UEM) 244*64kB (UEM) 190*128kB (UEM) 130*256kB (UEM) 77*512kB (UEM) 44*1024kB (EM) 17*2048kB (UEM) 12*4096kB (UM) = 253376kB
Node 0 Normal: 6759*4kB (UEM) 4044*8kB (UEM) 4085*16kB (UEM) 19244*32kB (UEM) 16700*64kB (UEM) 12866*128kB (UEM) 10839*256kB (UEM) 7294*512kB (UEM) 5079*1024kB (UEM) 1444*2048kB (UEM) 2709*4096kB (UEM) = 29219788kB
Node 1 Normal: 25995*4kB (UEM) 29980*8kB (UEM) 27256*16kB (UEM) 21986*32kB (UEM) 18650*64kB (UEM) 15434*128kB (UEM) 10997*256kB (UEM) 7625*512kB (UEM) 5589*1024kB (UEM) 1165*2048kB (UEM) 4778*4096kB (UEM) = 39051596kB
Node 2 Normal: 6272*4kB (UEM) 397*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 28264kB
Node 3 Normal: 2944*4kB (UE) 2060*8kB (UE) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 28256kB
Node 4 Normal: 124660*4kB (UEM) 115305*8kB (UEM) 64318*16kB (UEM) 36560*32kB (UEM) 18776*64kB (UEM) 11695*128kB (UEM) 8396*256kB (UEM) 10248*512kB (UEM) 4828*1024kB (UEM) 2*2048kB (EM) 0*4096kB = 18663032kB
Node 5 Normal: 10891*4kB (UEM) 38351*8kB (UEM) 33254*16kB (UEM) 27363*32kB (UEM) 13187*64kB (UEM) 20211*128kB (UEM) 17323*256kB (UEM) 11336*512kB (UEM) 7612*1024kB (UEM) 376*2048kB (UEM) 2439*4096kB (UEM) = 33982628kB
Node 6 Normal: 8911*4kB (UEM) 45632*8kB (UEM) 66318*16kB (UEM) 45797*32kB (UEM) 26527*64kB (UEM) 15376*128kB (UEM) 12129*256kB (UEM) 11610*512kB (UEM) 9321*1024kB (UEM) 1185*2048kB (UEM) 3512*4096kB (UEM) = 41999228kB
Node 7 Normal: 26452*4kB (UEM) 37175*8kB (UEM) 33131*16kB (UEM) 24352*32kB (UEM) 15725*64kB (UEM) 12285*128kB (UEM) 10374*256kB (UEM) 8533*512kB (UEM) 6608*1024kB (UEM) 1591*2048kB (UEM) 2311*4096kB (UEM) = 30806904kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB

上例中的标志位解释：

[MIGRATE_UNMOVABLE] = 'U',
[MIGRATE_RECLAIMABLE] = 'E',
[MIGRATE_MOVABLE]   = 'M',
[MIGRATE_RESERVE]   = 'R',
[MIGRATE_CMA]       = 'C',
[MIGRATE_ISOLATE]   = 'I',

分别表示不同size的内存链表中，所包含的内存类型（migrate_type）。之所以存在这个migrate_type，是为了方便内存迁移回收操作。

• 第4部分

第四部分是hugpage相关的信息。本案例中的机器上没有适用hugepage：

Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB

• 第5部分

第五部分是内存的pagecache和swap的使用情况。

539201 total pagecache pages
0 pages in swap cache
Swap cache stats: add 0, delete 0, find 0/0
Free swap  = 0kB
Total swap = 0kB
134192591 pages RAM
0 pages HighMem/MovableOnly
2318950 pages reserved

• 第6部分

最后一部分是按照进程级别输出的内存使用情况。

[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[ 1359]     0  1359    11108      489      23        0         -1000 systemd-udevd
[ 1986]     0  1986     7273      849      18        0             0 systemd-logind
[ 1993]    81  1993     6983      435      19        0          -900 dbus-daemon
[ 2037]     0  2037    25210      258      38        0             0 gssproxy
[ 2069]     0  2069     7274      237      19        0             0 atd
[ 3303]     0  3303    21422      647      45        0         -1000 sshd
[ 3306]     0  3306    21422      641      45        0         -1000 sshd
[ 3353]     0  3353     2417      654       9        0         -1000 iscsid
[ 3354]     0  3354     5945     4159      16        0         -1000 iscsid