记一次k8s修改docker存储路径引发的问题

Posted 2020-12-20

背景：

测试环境安装centos7的时候没有手动分区，自动分区根目录只有50G,而安装docker的时候也没有指定存储路径，默认是/var/lib/docker，我在/home底下创建一个目录docker_data/docker，然后将/var/lib/docker目录内容拷贝到/home/docker_data/docker目录，然后做软链接将/var/lib/docker指定到/home/docker_data/docker,分别将三个节点都这样做，刚开始做完都没有问题，后面有一次重启docker的时候有一个节点上面的pod就启动不起来了，使用kubectl describe pod 后得到以下信息

failed to inspect sandbox image "registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0": Error response from daemon: layer does not exist

分析：1、刚开始以为是这个镜像不存在了，使用docker pull重新拉取镜像，也提示拉取成功了

[root@node02 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
3.0: Pulling from google-containers/pause-amd64
Digest: sha256:3b3a29e3c90ae7762bdf587d19302e62485b6bef46e114b741f7d75dba023bd3
Status: Image is up to date for registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0

但是使用docker images查看没有任何镜像

[root@node02 ~]# docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

2、我以为是缓存的问题，使用docker rmi 想删除这个镜像，提示没有这个镜像

[root@node02 ~]# docker rmi registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
Error response from daemon: unrecognized image ID sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2

3、问别人说可能是缓存镜像层的问题，要清除缓存镜像层，于是我使用以下命令

[root@node02 ~]# docker system prune  -a
WARNING! This will remove:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated to them
  - all build cache

Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: bitnami/node-exporter:1.0.1-debian-10-r40
untagged: bitnami/node-exporter@sha256:93b7b6e1decabb3785af9d8a4638457264086eff0e39c88b33cb28a1bf5eb56c
deleted: sha256:bf3932c1a438bd5a370678277c55d6c0d96fc9b319da1a19bf7022542da97459
deleted: sha256:5457ebcdc2664ba370621a2747cbecbdda9f816f9bb372582445c6f07e942966
deleted: sha256:3b21c23574ea3a44256c9fd9ab20ccaa876fd19187ed8804a18d3f76cfc0380e
deleted: sha256:af30bbe16627da3074b8932259e2021f3560e1bedf69b5dd014a3b5d10f2c6bc
deleted: sha256:6f3a3e041f64bd68c124667d904bde76d95a57d596363ee4884ff938e6d4f471
deleted: sha256:5dd3dec5dec64a9944ed3e8d0fba5f209d5a91da387efb77aa96e1c68581de4a
deleted: sha256:049f84c2a771aef6f164f4e055aabdfcad2d173b5473f1277a47ff29e03e5a8d
untagged: busybox:1.28.3
untagged: busybox@sha256:58ac43b2cc92c687a32c8be6278e50a063579655fe3090125dcb2af0ff9e1a64
deleted: sha256:8ac48589692a53a9b8c2d1ceaa6b402665aa7fe667ba51ccc03002300856d8c7
deleted: sha256:0314be9edf00a925d59f9b88c9d8ccb34447ab677078874d8c14e7a6816e21e1

Total reclaimed space: 116.9MB

这个命令清除所有没有使用的镜像和容器，清除后仍然拉到不到镜像

4、大佬说可能还是缓存没有清理干净，于是我就删除以下目录

[root@node02 ~]# ls /home/docker_data/docker/image/overlay2/layerdb/

删除了，重启docker，再拉取，再docker images，仍然没有看到镜像

5、我干脆直接删除image目录

[root@node02 ~]# rm /home/docker_data/docker/image

再重启docker，再拉取镜像，再docker images，这回有镜像了，我再Kubectl delete pod删除相关pod，后面就正常重建Pod了