Address Sanitizer 调用了 OOM-killer
Posted
技术标签:
【中文标题】Address Sanitizer 调用了 OOM-killer【英文标题】：Address Sanitizier invokes OOM-killer 【发布时间】：2021-12-02 14:00:34 【问题描述】：我正在尝试使用 Address Sanitizer，但由于内存使用过多，内核不断杀死我的进程。如果没有 Address Sanitizer，该过程运行得很好。
该程序是使用 gcc-8.2.1 为 arm-v7a 编译的
-fno-omit-frame-pointer
-fsanitize=address
-fsanitize-recover=all
-fdata-sections
-ffunction-sections
-fPIC
我开始的过程如下：
ASAN_OPTIONS=debug=1:verbosity=0:detect_leaks=0:abort_on_error=0:halt_on_error=0:check_initialization_order=1:allocator_may_return_null=1 ./Launcher
有没有办法减少 Address Sanitizer 的内存占用？不幸的是，启用交换不是一种选择。
这是 dmesg 打印的内核日志：
[512792.413376] Launcher invoked oom-killer: gfp_mask=0x400dc0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), order=0, oom_score_adj=0
[512792.424695] CPU: 3 PID: 7786 Comm: Launcher Tainted: G W 5.4.1 #1
[512792.432821] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[512792.439455] Backtrace:
[512792.442039] [<8010eb1c>] (dump_backtrace) from [<8010eee0>] (show_stack+0x20/0x24)
[512792.449721] r7:811d32ec r6:00000000 r5:60070113 r4:811d32ec
[512792.455500] [<8010eec0>] (show_stack) from [<80ba06e8>] (dump_stack+0xbc/0xe8)
[512792.462840] [<80ba062c>] (dump_stack) from [<80257360>] (dump_header+0x64/0x440)
[512792.470343] r10:00000a24 r9:a9a4ce00 r8:00016f9c r7:80e82aac r6:a749fce0 r5:a9a4ce00
[512792.478275] r4:a749fce0 r3:6f25b167
[512792.481958] [<802572fc>] (dump_header) from [<80256364>] (oom_kill_process+0x494/0x4ac)
[512792.490066] r10:00000a24 r9:a9a4c100 r8:00016f9c r7:80e82aac r6:a749fce0 r5:a9a4ce00
[512792.497996] r4:a9a4d264
[512792.500636] [<80255ed0>] (oom_kill_process) from [<80256e8c>] (out_of_memory+0xf8/0x4ec)
[512792.508830] r10:00000a24 r9:a9a4c100 r8:00016f9c r7:8110b640 r6:8110b640 r5:811d8860
[512792.516760] r4:a749fce0
[512792.519405] [<80256d94>] (out_of_memory) from [<802a0910>] (__alloc_pages_nodemask+0xf7c/0x13a4)
[512792.528295] r9:00000000 r8:81107d30 r7:811d5588 r6:0000233c r5:00000000 r4:00000000
[512792.536153] [<8029f994>] (__alloc_pages_nodemask) from [<80285d10>] (__pte_alloc+0x34/0x1ac)
[512792.544697] r10:74b94000 r9:00000000 r8:00000000 r7:a8b9e580 r6:a8b9e580 r5:a7445d28
[512792.552628] r4:a7445d28
[512792.555271] [<80285cdc>] (__pte_alloc) from [<802869c8>] (copy_page_range+0x4ec/0x650)
[512792.563295] r9:00000000 r8:00000000 r7:a8b9e580 r6:a7174f4c r5:a8b9e580 r4:a7445d28
[512792.571148] [<802864dc>] (copy_page_range) from [<801241b8>] (dup_mm+0x470/0x4e0)
[512792.578736] r10:a7174f14 r9:a7174f10 r8:a8b9d680 r7:a7c36420 r6:a7174f4c r5:a8b9e580
[512792.586667] r4:a7835d20
[512792.589307] [<80123d48>] (dup_mm) from [<801255e0>] (copy_process+0x10bc/0x1888)
[512792.596807] r10:a749ff60 r9:ffffffff r8:00000000 r7:a749e000 r6:9d283400 r5:a825c300
[512792.604738] r4:00100000
[512792.607378] [<80124524>] (copy_process) from [<80125fb8>] (_do_fork+0x90/0x750)
[512792.614792] r10:00100000 r9:a749e000 r8:801011c4 r7:a749e000 r6:a749ff60 r5:6f25b167
[512792.622722] r4:00000001
[512792.625362] [<80125f28>] (_do_fork) from [<80126954>] (sys_clone+0x80/0x9c)
[512792.632428] r10:00000078 r9:a749e000 r8:801011c4 r7:00000078 r6:7649e000 r5:6f25b167
[512792.640358] r4:a749e000
[512792.643001] [<801268d4>] (sys_clone) from [<80101000>] (ret_fast_syscall+0x0/0x28)
[512792.650671] Exception stack(0xa749ffa8 to 0xa749fff0)
[512792.655828] ffa0: 54ad00fc 76ffe964 00100011 00000000 54ad00fc 00000000
[512792.664112] ffc0: 54ad00fc 76ffe964 7649e000 00000078 54ad0100 54ad0120 00000001 54ad0280
[512792.672391] ffe0: 00000078 54ad00e8 763d590b 763bf746
[512792.677546] r5:76ffe964 r4:54ad00fc
[512792.681484] Mem-Info:
[512792.683936] active_anon:158884 inactive_anon:15315 isolated_anon:0
active_file:1041 inactive_file:1140 isolated_file:0
unevictable:2224 dirty:8 writeback:1 unstable:0
slab_reclaimable:4553 slab_unreclaimable:4490
mapped:5064 shmem:17635 pagetables:1579 bounce:0
free:56987 free_pcp:173 free_cma:53962
[512792.718450] Node 0 active_anon:635536kB inactive_anon:61260kB active_file:4264kB inactive_file:5460kB unevictable:8896kB isolated(anon):0kB isolated(file):0kB mapped:21056kB dirty:32kB writeback:4kB shmem:70540kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[512792.742142] Normal free:226708kB min:3312kB low:4140kB high:4968kB active_anon:635436kB inactive_anon:61260kB active_file:4584kB inactive_file:5652kB unevictable:8896kB writepending:36kB present:1048576kB managed:1015668kB mlocked:0kB kernel_stack:1216kB pagetables:6316kB bounce:0kB free_pcp:192kB local_pcp:0kB free_cma:215848kB
[512792.771461] lowmem_reserve[]: 0 0 0
[512792.775161] Normal: 1651*4kB (UMEC) 839*8kB (UMEC) 495*16kB (UMEC) 221*32kB (UMEC) 78*64kB (UEC) 29*128kB (MC) 1*256kB (U) 40*512kB (C) 35*1024kB (C) 21*2048kB (C) 10*4096kB (C) 2*8192kB (C) 0*16384kB 1*32768kB (C) = 226708kB
[512792.795442] 20243 total pagecache pages
[512792.799391] 0 pages in swap cache
[512792.802816] Swap cache stats: add 0, delete 0, find 0/0
[512792.808232] Free swap = 0kB
[512792.811225] Total swap = 0kB
[512792.814296] 262144 pages RAM
[512792.817288] 0 pages HighMem/MovableOnly
[512792.821232] 8227 pages reserved
[512792.824558] 81920 pages cma reserved
[512792.828247] Tasks state (memory values in pages):
[512792.833057] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[512792.841890] [ 211] 0 211 9965 1608 67584 0 0 systemd-journal
[512792.851149] [ 224] 0 224 3848 249 16384 0 -1000 systemd-udevd
[512792.860222] [ 317] 0 317 1559 339 12288 0 0 dhclient
[512792.868867] [ 316] 0 316 1559 348 14336 0 0 dhclient
[512792.877508] [ 333] 0 333 1810 856 14336 0 0 haveged
[512792.886061] [ 334] 101 334 4985 261 22528 0 0 systemd-timesyn
[512792.895309] [ 336] 104 336 1342 167 12288 0 0 rpcbind
[512792.903866] [ 368] 106 368 1333 218 12288 0 -900 dbus-daemon
[512792.912684] [ 369] 0 369 6193 356 22528 0 0 rsyslogd
[512792.921327] [ 370] 0 370 2681 178 18432 0 0 systemd-logind
[512792.930490] [ 372] 0 372 1625 158 14336 0 0 cron
[512792.938784] [ 431] 0 431 428 122 10240 0 0 motion_sensor
[512792.947870] [ 560] 0 560 8756 207 18432 0 0 automount
[512792.956597] [ 564] 0 564 1190 172 12288 0 0 login
[512792.964988] [ 566] 0 566 1338 98 12288 0 0 agetty
[512792.973372] [ 572] 0 572 2218 276 16384 0 -1000 sshd
[512792.981664] [ 574] 0 574 946 33 12288 0 0 inputattach
[512792.990569] [ 637] 0 637 3017 379 18432 0 0 systemd
[512792.999122] [ 640] 0 640 3504 402 20480 0 0 (sd-pam)
[512793.007768] [ 653] 0 653 1760 329 12288 0 0 bash
[512793.016057] [ 671] 0 671 2599 1116 18432 0 0 Server.
[512793.025310] [ 732] 0 732 1300 132 12288 0 0 dbus-daemon
[512793.034212] [ 31836] 0 31836 3173 980 22528 0 0 sshd
[512793.042428] [ 31847] 0 31847 422 154 8192 0 0 sftp-server
[512793.051332] [ 5350] 0 5350 2555 351 16384 0 0 sshd
[512793.059631] [ 5452] 0 5452 1793 379 16384 0 0 bash
[512793.067924] [ 5823] 0 5823 2555 350 16384 0 0 sshd
[512793.076216] [ 5833] 0 5833 1760 326 14336 0 0 bash
[512793.084509] [ 6822] 0 6822 792 31 10240 0 0 xinit
[512793.092813] [ 6823] 0 6823 29526 5386 112640 0 0 Xorg
[512793.101103] [ 6827] 0 6827 3655 866 22528 0 0 xterm
[512793.109488] [ 6829] 0 6829 1620 114 14336 0 0 bash
[512793.117784] [ 7256] 0 7256 1549 322 12288 0 0 watch
[512793.126169] [ 7363] 0 7363 127832 56725 520192 0 0 gdb
[512793.134370] [ 7368] 0 7368 281561 93707 1046528 0 0 Launcher
[512793.143613] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),task=Launcher,pid=7368,uid=0
[512793.152974] Out of memory: Killed process 7368 (Launcher) total-vm:1126244kB, anon-rss:365128kB, file-rss:5700kB, shmem-rss:4000kB, UID:0 pgtables:1046528kB oom_score_adj:0
[512793.387824] oom_reaper: reaped process 7368 (Launcher), now anon-rss:0kB, file-rss:0kB, shmem-rss:4000kB
【问题讨论】：
查找是否可以禁用缓冲区溢出检查（它会消耗大量内存 AFAIK）。 @MarekR 我找不到任何 ASAN_OPTION 来禁用缓冲区溢出检查。你能指出我正确的方向吗？ 【参考方案1】：您可以减少一些 Asan 功能（或在单独运行中一一启用）：
# Disable UAR error detection (reduces code and heap size)
CFLAGS+='-fsanitize-address-use-after-return=never -fno-sanitize-address-use-after-scope'
export ASAN_OPTIONS="$ASAN_OPTIONS:detect_stack_use_after_return=1"
# Disable inline instrumentation (slower but saves code size)
CFLAGS+='-fsanitize-address-outline-instrumentation'
# Reduce heap quarantine (reduces heap consumption but also lowers chance of UAF detection)
export ASAN_OPTIONS="$ASAN_OPTIONS:quarantine_size_mb=16"
# Do not keep full backtrace of malloc origin (slightly complicates debugging but reduces heap size)
export ASAN_OPTIONS="$ASAN_OPTIONS:malloc_context_size=5"
编译器选项适用于 Clang，但 GCC 也有类似的开关。
至于交换，我们在 RAM 中启用 compressed swap 有很好的经验。
【讨论】：
太好了，非常感谢。 :)以上是关于Address Sanitizer 调用了 OOM-killer的主要内容，如果未能解决你的问题，请参考以下文章
Android jni 集成 Address Sanitizer
Android jni 集成 Address Sanitizer