删除K8S集群的/var/lib/kubelet目录报Device or resource busy错误解决方法

Posted 2023-05-04

参考技术A 无法删除/var/lib/kubelet目录

查看挂载信息

umount这些挂在，然后再删除

rm -rf /var/lib/docker 设备或资源忙问题处理

在安装部署微服务时，在不同实例机器上需要安装部署修改docker默认路径，重新映射到新的磁盘空间，需要删除默认安装路径，有时会碰到无法正常删除，如下：
rm -rf /var/lib/docker 设备或资源忙

[root@test-master playbooks]# rm -rf /var/lib/docker
rm: 无法删除"/var/lib/docker/containers/3bee81da15b9cf5a17b5ddbd3f777a3bcacb26f3f9e65f500aee86982c480342/shm": 设备或资源忙
rm: 无法删除"/var/lib/docker/containers/129b31e3f4663219279832a925cbb656339c48d204593e2e74e79e4574a7851a/shm": 设备或资源忙
rm: 无法删除"/var/lib/docker/containers/3033a2d03057ef17612139e998d772047b295aa46a022d3944c12c0d93097a3c/shm": 设备或资源忙

问题排查方式如下：
查看docker目前状态
[root@test-master playbooks]# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─custom.conf
Active: active (running) since 二 2019-12-24 14:19:53 CST; 19h ago
Docs: http://docs.docker.com
Main PID: 11695 (dockerd-current)
CGroup: /system.slice/docker.service
├─11490 /usr/bin/docker-containerd-shim-current f01c5fa965a6627e1b463b13ee8bea3e8a7af6c07653842c5730b1d2c31f000b /var/...
├─11695 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=d...
├─11704 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-in...
├─14828 /usr/bin/docker-containerd-shim-current 3033a2d03057ef17612139e998d772047b295aa46a022d3944c12c0d93097a3c /var/...
├─24024 /usr/bin/docker-containerd-shim-current 3bee81da15b9cf5a17b5ddbd3f777a3bcacb26f3f9e65f500aee86982c480342 /var/...
├─24042 /usr/bin/docker-containerd-shim-current 129b31e3f4663219279832a925cbb656339c48d204593e2e74e79e4574a7851a /var/...
└─24186 /usr/bin/docker-containerd-shim-current 8e75a3776132d00ac5834dce9c69a00954f213d0986ab44245f053649b821328 /var/...

接下来看如果停止docker服务运行在删除如果成功，那就算OK
如果还是提示设备忙，如下：
[root@test-master playbooks]# systemctl stop docker
[root@test-master playbooks]# rm -rf /var/lib/docker
rm: 无法删除"/var/lib/docker/containers/3bee81da15b9cf5a17b5ddbd3f777a3bcacb26f3f9e65f500aee86982c480342/shm": 设备或资源忙
rm: 无法删除"/var/lib/docker/containers/129b31e3f4663219279832a925cbb656339c48d204593e2e74e79e4574a7851a/shm": 设备或资源忙
rm: 无法删除"/var/lib/docker/containers/3033a2d03057ef17612139e998d772047b295aa46a022d3944c12c0d93097a3c/shm": 设备或资源忙
rm: 无法删除"/var/lib/docker/overlay2/c01587998e5dd5ca2e51b3414f528367234615d9c48f589e7abf3b03394c798a/merged": 设备或资源忙
rm: 无法删除"/var/lib/docker/overlay2/e28014890386bc9a1b5824daa46a6b5085db99db981e1e86ca77669e6670215e/merged": 设备或资源忙
rm: 无法删除"/var/lib/docker/overlay2/91bac1b5198296322b23eafd80a58503a2f0ef1d05802483e02b726384bc3991/merged": 设备或资源忙
rm: 无法删除"/var/lib/docker/overlay2/4ddb10bbce9c009aabbadc958c17b81682b74660e167384c8fa32d512a87413d/merged": 设备或资源忙

如上问题提示说明，containers、overlay2这个容器文件需要重新挂载才行，如下查看
[root@test-master ~]# ls -l /var/lib/docker
总用量 0
drwx------. 2 root root 6 12月 25 10:23 containers
drwx------. 2 root root 6 12月 25 10:23 overlay2

[root@test-master ~]#
[root@test-master ~]# umount /var/lib/docker/containers/
[root@test-master ~]# umount /var/lib/docker/overlay2/

在删除即可。
[root@test-master ~]# rm -rf /var/lib/docker
[root@test-master ~]#
[root@test-master ~]#