Docker load 大镜像(17G) 报错no space left on device

Posted 2022-10-23 琦彦

# 导入加载镜像，test.tar大小有16G左右
docker load -i test.tar

报错信息

制作镜像失败：【ApplyLayer exit status1 stdout:stderr:write /home/appuser/.local/MIb/python3.7/site-
packages/scipy/linalg/tests/pycache_/test_decomp_update.cpython-37.pyc:no space left on device

问题排查

是Base Device Size的问题，其默认值是10G，而容器经过一段时间的运行写入文件量早已超过10G，就造成无法写入数据的问题了

Doker 的默认配置

# docker info
[root@fly ~]# docker info
Containers: 8
 Running: 8
 Paused: 0
 Stopped: 0
Images: 8
Server Version: 18.09.0
Storage Driver: devicemapper
 Pool Name: docker-0:108-164868621-pool
 Pool Blocksize: 65.54kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Udev Sync Supported: true
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data loop file: /public/docker/devicemapper/devicemapper/data
 Metadata loop file: /public/docker/devicemapper/devicemapper/metadata
 Data Space Used: 846.6MB
 Data Space Total: 107.4GB
 Data Space Available: 106.5GB
 Metadata Space Used: 19.81MB
 Metadata Space Total: 17.05GB
 Metadata Space Available: 17.03GB
 Thin Pool Minimum Free Space: 10.74GB
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.149-RHEL7 (2018-07-20)
Logging Driver: json-file
Cgroup Driver: systemd
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: c4446665cb9c30056f4998ed953e6d4ff22c7c39
runc version: 4fc53a81fb7c994640722ac585fa9ca548971871
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-957.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 188.2GiB
Name: admin1
ID: 5IMT:GNNU:MFJS:IJJ6:RZJS:7CA7:N5QC:ZSF6:223N:5CB6:NFFM:W2N6
Docker Root Dir: /public/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 192.168.172.128:5000
 192.168.172.128:5001
 127.0.0.0/8
Registry Mirrors:
 http://hub-mirror.c.163.com/
 https://registry.docker-cn.com/
 https://docker.mirrors.ustc.edu.cn/
Live Restore Enabled: false
Product License: Community Engine

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
         Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

• Base Device Size: 10.74GB
• Data Space Total: 107.4GB
• Metadata Space Total: 17.05GB

问题解决

调整docker服务端启动参数

# vim /usr/lib/systemd/system/docker.service
[root@fly ~]# grep "ExecStart=" /usr/lib/systemd/system/docker.service
ExecStart=/usr/bin/dockerd  -H fd:// --containerd=/run/containerd/containerd.sock -s=devicemapper --storage-opt dm.basesize=60G --storage-opt dm.loopmetadatasize=20G

• dm.basesize 默认为10G，限制容器和镜像的大小
• dm.loopdatasize 存储池大小，默认为100G
• dm.loopmetadatasize 元数据大小，默认为2G
• dm.datadev 存储池设备，默认生成一个/var/lib/docker/devicemapper/devicemapper/data文件
• dm.metadatadev 元数据设备，默认生成一个/var/lib/docker/devicemapper/devicemapper/metadata文件
• dm.fs 用于基本映像的文件系统（xfs或ext4）
• dm.blocksize：精简池的自定义块大小。默认值为64K。

具体可参考官方文档： https://docs.docker.com/engine/reference/commandline/dockerd/

修改完配置后, 重启docker

systemctl daemon-reload
systemctl restart docker

查看调整

# ps -ef | grep docker|grep /usr/bin/dockerd|grep -v grep
[root@fly ~]# ps -ef | grep docker|grep /usr/bin/dockerd|grep -v grep
root     23665     1  9 10月12 ?      01:25:52 /usr/bin/dockerd --bip=173.0.52.1/24 --ip-masq=true --mtu=1450 --graph /public/docker --storage-opt dm.basesize
=60G --storage-opt dm.loopmetadatasize=20G

参考

https://docs.docker.com/engine/reference/commandline/dockerd/

dm.basesize
=60G --storage-opt dm.loopmetadatasize=20G

##  参考

https://docs.docker.com/engine/reference/commandline/dockerd/