Docker中使用createdump调试coreclr

Posted 2020-12-30 gool

应用上线后可能出现一些问题，通过源码排查，日志分析都不能确定具体原因的情况下，可以使用dump转存文件分析，netcore对于linux系统dump提供了createdump工具，配合lldb sos插件进行clr级分析

启动一个dotnet容器，使用特权运行，createdump要访问其他进程内存，特权是必须的

docker run --name dotnet --rm -it --privileged=true microsoft/dotnet /bin/bash

创建并运行一个webapi示例

cd ~/
dotnet new webapi
dotnet run > /dev/null 2>&1 &

查找dotnet中的createdump工具路径

find /usr/share -name createdump

列出所有dotnet进程

ps -al | grep dotnet

创建dump文件

/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.2/createdump PID
ls /tmp/

将dump文件复制到本机，因为docker不支持容器之间复制，需要中转

docker cp dotnet:/tmp/coredump.324 ./

启动一个lldb容器

docker run --name lldb --rm -it -v ./:var/dump/ daocloud.io/ss22219/dotnet-lldb /bin/bash

复制dump文件到容器，分析dump文件

docker cp coredump.324 lldb:/root
lldb coredump.324

ss22219/dotnet-lldb镜像开源在github上，将lldb集成到dotnet sdk中，并且自动加载sos插件
使用方法是lldb dumpfile，sos的命令列表如下

SOS 命令列表

输入 "soshelp <functionname>" 查看函数详情

Object Inspection                  Examining code and stacks
-----------------------------      -----------------------------
DumpObj (dumpobj)                  Threads (clrthreads)
DumpArray                          ThreadState
DumpStackObjects (dso)             IP2MD (ip2md)
DumpHeap (dumpheap)                u (clru)
DumpVC                             DumpStack (dumpstack)
GCRoot (gcroot)                    EEStack (eestack)
PrintException (pe)                ClrStack (clrstack)
                                   GCInfo
                                   EHInfo
                                   bpmd (bpmd)

Examining CLR data structures      Diagnostic Utilities
-----------------------------      -----------------------------
DumpDomain                         VerifyHeap
EEHeap (eeheap)                    FindAppDomain
Name2EE (name2ee)                  DumpLog (dumplog)
DumpMT (dumpmt)                    CreateDump (createdump)
DumpClass (dumpclass)
DumpMD (dumpmd)
Token2EE
DumpModule (dumpmodule)
DumpAssembly
DumpRuntimeTypes
DumpIL (dumpil)
DumpSig
DumpSigElem

Examining the GC history           Other
-----------------------------      -----------------------------
HistInit (histinit)                FAQ
HistRoot (histroot)                Help (soshelp)
HistObj  (histobj)
HistObjFind (histobjfind)
HistClear (histclear)

别名列表

bpmd            -> sos bpmd
clrstack        -> sos ClrStack
clrthreads      -> sos Threads
clru            -> sos U
createdump      -> sos CreateDump
dso             -> sos DumpStackObjects
dumpclass       -> sos DumpClass
dumpheap        -> sos DumpHeap
dumpil          -> sos DumpIL
dumplog         -> sos DumpLog
dumpmd          -> sos DumpMD
dumpmodule      -> sos DumpModule
dumpmt          -> sos DumpMT
dumpobj         -> sos DumpObj
dumpstack       -> sos DumpStack     
eeheap          -> sos EEHeap
eestack         -> sos EEStack
gcroot          -> sos GCRoot
histinit        -> sos HistInit
histroot        -> sos HistRoot
histobj         -> sos HistObj
histobjfind     -> sos HistObjFind
histclear       -> sos HistClear
ip2md           -> sos IP2MD
name2ee         -> sos Name2EE
pe              -> sos PrintException
soshelp         -> sos Help

相关资料：
https://github.com/dotnet/coreclr/blob/master/Documentation/building/debugging-instructions.md
https://github.com/dotnet/coreclr/blob/master/Documentation/botr/xplat-minidump-generation.md