Using qemu to debug kernel (by quqi99)

Posted 2022-08-02 quqi99

作者：张华 发表于：2021-12-06
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
( http://blog.csdn.net/quqi99 )

五年前做过类似的事，见: https://blog.csdn.net/quqi99/article/details/50640902

Prepare Kernel

sudo apt install qemu qemu-kvm bridge-utils virt-manager -y
sudo apt install bison flex libelf-dev libssl-dev build-essential libncurses-dev -y
sudo apt install libncurses5-dev libncursesw5-dev -y
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
cd linux-2.6
#git checkout -b v5.9 v5.9
make defconfig
#cp -v /boot/config-$(uname -r) .config
./scripts/config -e DEBUG_INFO -e GDB_SCRIPTS
make menuconfig
make -j $(nproc) bzImage
#sudo make modules_install
#sudo make install

./scripts/config -m E1000
make SUBDIRS=drivers/net/ethernet/intel/e1000 modules
ls drivers/net/ethernet/intel/e1000/e1000.ko

Prepare Root fs

# Way 1
#mv rootfs.img ./core/rootfs.cpio.gz 
#cd core
#gunzip rootfs.cpio.gz
#cpio -idmv < rootfs.cpio
#rm -rf rootfs.cpio
# Way 2
#sudo apt-get install debootstrap
#wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-image.sh -O create-image.sh
#chmod +x create-image.sh
#./create-image.sh
# Way 3
cd /bak/work/linux/rootfs
wget http://cdimage.ubuntu.com/cdimage/ubuntu-base/releases/20.04/release/ubuntu-base-20.04.1-base-amd64.tar.gz
dd if=/dev/zero of=rootfs.img bs=1024 count=1M
mkfs.ext4 -F -L linuxroot rootfs.img
sudo mkdir -p /mnt/tmpdir && sudo chown -R $USER /mnt/tmpdir
sudo mount -o loop rootfs.img /mnt/tmpdir/
sudo tar zxvf ubuntu-base-20.04.1-base-amd64.tar.gz -C /mnt/tmpdir/
sudo cp /etc/resolv.conf /mnt/tmpdir/etc/
sudo mount -t proc /proc /mnt/tmpdir/proc
sudo mount -t sysfs /sys /mnt/tmpdir/sys
sudo mount -o bind /dev /mnt/tmpdir/dev
sudo mount -o bind /dev/pts /mnt/tmpdir/dev/pts
sudo chroot /mnt/tmpdir
apt update
apt-get install language-pack-en-base sudo ssh net-tools ethtool wireless-tools ifupdown \\
  network-manager iputils-ping rsyslog htop vim xinit xorg alsa-utils --no-install-recommends
useradd ubuntu
passwd ubuntu
echo "debug" > /etc/hostname
cat << EOF | sudo tee -a /etc/hosts
127.0.0.1 localhost
127.0.0.1 debug
EOF
echo "allowed_users=anybody" > /etc/X11/Xwrapper.config
dpkg-reconfigure tzdata
exit
sudo umount /mnt/tmpdir/proc/
sudo umount /mnt/tmpdir/sys/
sudo umount /mnt/tmpdir/dev/pts/
sudo umount /mnt/tmpdir/dev/
sudo umount /mnt/tmpdir/

或:

#create rootfs
sudo apt-get install debootstrap
wget https://raw.githubusercontent.com/google/syzkaller/master/tools/create-image.sh -O create-image.sh
chmod +x create-image.sh
./create-image.sh
#sudo mkdir -p /mnt/chroot
#sudo mount -o loop stretch.img /mnt/chroot
#sudo cp -a chroot/. /mnt/chroot/.
#sudo umount /mnt/chroot
sudo ip tuntap add mod tap user $(whoami)
sudo tunctl -t tap0 -u root    
sudo brctl addif virbr0 tap0
sudo ifconfig tap0 0.0.0.0
sudo brctl showstp virbr0
sudo qemu-system-x86_64 \\
    -m 2G \\
    -smp 2 \\
    -kernel /boot/vmlinuz-5.15.0-25-generic \\
    -append "nokaslr console=ttyS0 root=/dev/sda earlyprintk=serial" \\
    -drive file=/bak/work/kernel/stretch.img,format=raw \\
    -net nic \\
    -net tap,ifname=tap0,script=no,downscript=no \\
    -enable-kvm \\
    -nographic \\
    -pidfile vm.pid \\
    2>&1 | tee vm.log

Debug Kernel

sudo qemu-system-x86_64 -s -S -m 1024M -smp 4 \\
  -kernel /bak/linux/linux-2.6/arch/x86/boot/bzImage \\
  -append "nokaslr root=/dev/sda console=ttyS0" \\
  -drive file=/bak/work/linux/rootfs/rootfs.img,format=raw,index=0,media=disk \\
  --enable-kvm -cpu host -nographic

cgdb /bak/linux/linux-2.6/vmlinux
(gdb) set architecture i8086
(gdb) target remote :1234
(gdb) hbreak start_kernel
(gdb) b start_kernel
(gdb) c
(gdb) bt
(gdb) list
# Inside VM, echo 'c' | sudo tee /proc/sysrq-trigger
(gdb) add-symbol-file vmlinux 0xffffffff81000000 #echo 0x$(sudo cat /proc/kallsyms | egrep -e "T _text$" | awk 'print $1')
(gdb) b sysrq_handle_crash

Use initramfs instead

cd /bak/work/linux/busybox
wget https://busybox.net/downloads/busybox-1.34.1.tar.bz2
tar -xf busybox-1.34.1.tar.bz2
cd busybox-1.34.1
make menuconfig
# Settings  ---> [*] Build static binary (no shared libs)
make -j$(nproc) && make install

cd _install/
mkdir proc sys
cat << EOF | tee init
#!/bin/sh
echo "==DBG== INIT SCRIPT"
mkdir /tmp
mount -t proc none /proc
mount -t sysfs none /sys
mount -t debugfs none /sys/kernel/debug
mount -t tmpfs none /tmp
mdev -s
echo -e "==DBG== Boot took $(cut -d' ' -f1 /proc/uptime) seconds"
setsid /bin/cttyhack setuidgid 1000 /bin/sh
EOF
chmod +x init
find . | cpio -o --format=newc > ./rootfs.img

qemu-system-x86_64 -kernel /bak/linux/linux-2.6/arch/x86/boot/bzImage \\
  -initrd  /bak/work/linux/busybox/busybox-1.34.1/_install/rootfs.img \\
  -append "nokaslr console=ttyS0" -nographic -s -S
cgdb /bak/linux/linux-2.6/vmlinux
(gdb) target remote :1234

Failed to use eclipse due to ‘Failed to execute MI command’

# download eclipse PlatformRuntime version
# https://archive.eclipse.org/eclipse/downloads/drops4/R-4.21-202109060500/#PlatformRuntime
# As of 2017, Goclipse is no longer actively maintained, We use GoSublime instead of eclipse for Go
# Install Pydev - http://www.pydev.org/updates
# Install CDT(https://www.eclipse.org/cdt/downloads.php) - https://download.eclipse.org/tools/cdt/releases/10.4

# use cdt to debug kernel
Import existing C/C++ Project -> Makefile Project with Existing Code, select /bak/linux/linux-2.6
Run -> Debug configurations -> C/C++ Attach to Application -> Right click to create,

in 'Debugger' TAB
Debugger: gdbserver
Type: TCP
Host name or IP address: node1
Port number: 1234

in 'Main' TAB
select '/bak/linux/linux-2.6/vmlinux' as C/C++ Application
select 'Disable auto build

Failed to execute MI command:
settings --> Debugger --> Main Tab --> Browse to gdb compiled for the same platform
/usr/bin/gdb

20220629更新　- 　编译内核

• ubuntu kernel一般是用fakeroot来编译(https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel), 它编译的模块不会那么多不会遇到mkinitramfs慢的问题，也不会遇到key的问题(unsigned)
• upstream kernel如果不用make localmodconfig会编译所有模块，这样在运行mkinitramfs时会奇慢无比
• 如果遇到mkinitramfs奇慢无比，且grub又无法进，可直接指定grub项来回到原内核版本（GRUB_DEFAULT=“1>4”），１指/boot/grub/grub.cfg中的submenu项，4指submenu中的menuentry项(下标从0开始)

sudo apt-get install libncurses5-dev openssl libssl-dev build-essential openssl pkg-config libc6-dev \\
  bison flex libelf-dev minizip libidn11-dev libidn11 libssl-dev fakeroot zlibc \\
  makedumpfile kernel-wedge libelf-dev asciidoc binutils-dev ncurses-dev qttools5-dev liblz4-tool -y
#Speed up subsequent compilation
sudo apt install ccache -y
export USE_CCACHE=1
export CCACHE_DIR="/home/hua/.ccache" 
export CC="ccache gcc"    
export CXX="ccache g++"    
export PATH="$PATH:/usr/lib/ccache"
ccache -M 2G
sudo cp /usr/bin/ccache /usr/local/bin/
sudo ln -s ccache /usr/local/bin/gcc
sudo ln -s ccache /usr/local/bin/g++
sudo ln -s ccache /usr/local/bin/cc
sudo ln -s ccache /usr/local/bin/c++
ccache -s

cd /bak/kernel
#sudo apt source linux-image-unsigned-$(uname -r)
#sudo apt-get source linux-image-$(uname -r)
#sudo apt source linux
sudo apt-get install linux-source
sudo tar -xf linux-source-5.15.0.tar.bz2
/bak/kernel/linux-source-5.15.0
sudo chown -R $USER /bak/kernel/
cp /boot/config-$(uname -r) .config
sed -i 's/CONFIG_android_BINDER_IPC=m/CONFIG_ANDROID_BINDER_IPC=y/g' .config
sed -i 's/CONFIG_ANDROID_BINDERFS=m/CONFIG_ANDROID_BINDERFS=y/g' .config
sed -i 's/CONFIG_ANDROID_BINDER_DEVICES=""/CONFIG_ANDROID_BINDER_DEVICES="binder,hwbinder,vndbinder"/g' .config\\
#make localmodconfig   #speed compliling time for test
#make defconfig
#for the first compilation, should first click 'Load', then 'Save'
make menuconfig
cp .config /tmp/ && make mrproper && cp /tmp/.config .
grep -i -r 'binder' .config

#then use vim to modify .config, or set CONFIG_SYSTEM_TRUSTED_KEYRING=n (scripts/config --disable SYSTEM_TRUSTED_KEYS)
#
# Certificates for signature checking
#
CONFIG_MODULE_SIG_KEY="certs/signing_key.pem"
CONFIG_SYSTEM_TRUSTED_KEYRING=y
CONFIG_SYSTEM_TRUSTED_KEYS=""
CONFIG_SYSTEM_EXTRA_CERTIFICATE=y
CONFIG_SYSTEM_EXTRA_CERTIFICATE_SIZE=4096
CONFIG_SECONDARY_TRUSTED_KEYRING=y
CONFIG_SYSTEM_BLACKLIST_KEYRING=y
CONFIG_SYSTEM_BLACKLIST_HASH_LIST=""
CONFIG_SYSTEM_REVOCATION_LIST=y
CONFIG_SYSTEM_REVOCATION_KEYS=""
# end of Certificates for signature checking

#make bzImage -j$(nproc)
#make modules -j$(nproc)
#make -j$(nproc)
#make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- modules
make -j1
sudo make modules_install

#这里出错实际上因为之前空间不足删除了/usr/src所致，/var/lib/dkms/nvidia里有软链指向/usr/src
#Error! Could not locate dkms.conf file
for i in /var/lib/dkms/*/[^k]*/source; do [ -e "$i" ] || echo "$i";done
sudo cp -r /var/lib/dkms/backport-iwlwifi ~/dkms_bakup/
sudo cp -r /var/lib/dkms/nvidia ~/dkms_bakup/
sudo cp -r /var/lib/dkms/sysdig ~/dkms_bakup/
sudo cp -r /var/lib/dkms/v4l2loopback ~/dkms_bakup/
sudo rm -rf /var/lib/dkms/backport-iwlwifi
sudo rm -rf /var/lib/dkms/nvidia
sudo rm -rf /var/lib/dkms/sysdig
sudo rm -rf /var/lib/dkms/v4l2loopback

#sudo make install
sudo mkinitramfs -v /lib/modules/5.15.35 -o /boot/initrd.img-5.15.35
sudo cp /bak/kernel/linux-source-5.15.0/arch/x86/boot/bzImage /boot/vmlinuz-5.15.35
sudo cp /bak/kernel/linux-source-5.15.0/System.map /boot/System.map-5.15.35
sudo update-grub2
shutdown -r now

下面改为fakeroot来build ubuntu kernel

#https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
apt source linux-image-unsigned-$(uname -r)
cd linux-5.15.0
chmod a+x debian/rules
chmod a+x debian/scripts/*
chmod a+x debian/scripts/misc/*
LANG=C fakeroot debian/rules clean
#可按/来搜索配置找到配置路径，如搜索CONFIG_ANDROID_BINDER_DEVICES
LANG=C fakeroot debian/rules editconfigs
#将下列配置：
CONFIG_ANDROID_BINDER_IPC=y
CONFIG_ANDROID_BINDERFS=y
CONFIG_ANDROID_BINDER_DEVICES="binder,hwbinder,vndbinder"
CONFIG_FUSE_FS=y
#改成:
CONFIG_ANDROID_BINDER_IPC=m
CONFIG_ANDROID_BINDERFS=m
CONFIG_ANDROID_BINDER_DEVICES=""
CONFIG_FUSE_FS=y
#注意，为了避免再次运行editconfigs时验证出错，可能也要同时修改debian.master/config/annotations与debian.master/config/amd64/config.flavour.generic

sudo apt-get install libncurses-dev gawk flex bison openssl libssl-dev dkms libelf-dev libudev-dev libpci-dev libiberty-dev autoconf devscripts default-jdk
sudo apt-get build-dep linux linux-image-$(uname -r)
# quicker build:
LANG=C fakeroot debian/rules DEB_BUILD_OPTIONS=parallel=4 binary-headers binary-generic binary-perarch
#we should always use 'skipabi=true', 'skipmodule=true' means only generate linux-image and linux-header
# if you need linux-tools or lowlatency kernel, run instead:
LANG=C fakeroot debian/rules binary

如果报这下面错,是要安装jdk(sudo apt install default-jdk), 但怎么不重新编译只局部编译kernel-tools呢?

install: cannot stat '/bak/kernel/linux-5.15.0/debian/build/tools-perarch/tools/perf/libperf-jvmti.so': No such file or directory

对于下面的’ NOEXTRAS=1 skipabi=true skipmodule=true’只重建debs，但没有修改过的源文件被重建。因为debian/rules不是内核的Makefile。它无法知道你编辑的文件是最终内核的依赖关系，因为这些依赖关系在真正的Makefile中。见: http://cn.voidcc.com/question/p-neprwdjp-ou.html
注: 经找专家确认, ubuntu就是无法编译局部部件的, 只能整个地再编译一次. 当然可以类似make -C tools/perf来编译二进制测试, 但那是无法编译成deb包的.

#LANG=C NOEXTRAS=1 skipabi=true skipmodule=true fakeroot debian/rules DEB_BUILD_OPTIONS=parallel=4 binary-headers binary-generic binary-perarch

要创建符号表的话再添加:skipdbg=false

安装kernel后如果nvidia-smi不work的话可能需要重装nvidia

sudo apt install nvidia-driver-418 -y
$ ls /var/lib/dkms/nvidia
470.129.06
$ dkms status
nvidia/470.129.06: added

Reference

[1] https://www.jeanleo.com/2020/08/30/qemu%E8%B0%83%E8%AF%95%E5%86%85%E6%A0%B8%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA/
[2] https://cloud.tencent.com/developer/article/1793157
[3] https://cloud.tencent.com/developer/article/1793157
[4] https://gist.github.com/adwait1-G/5b4ae85239619e6f94b01cfb1bdb7dd6
[5] http://nickdesaulniers.github.io/blog/2018/10/24/booting-a-custom-linux-kernel-in-qemu-and-debugging-it-with-gdb/