docker cp 源码分析

Posted 2021-07-23 石头的安全料理屋

本文编写时，最新release为v20.10.6, 因此代码均为v20.10.6分支的代码

1. docker cp简介

在容器和宿主机的文件系统直接复制文件。执行cp命令的入口有两个，分别是docker container cp和docker cp, 两者作用相同， 后者可能在未来会被删除。

https://docs.docker.com/engine/reference/commandline/container_cp/

2. 源码入口位置

由cli接收cp命令参数,发送至docker engine api

cli与engine api的代码入口分别位于:

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cp.go#L43

https://github.com/moby/moby/blob/v20.10.6/api/server/router/container/copy.go

3. docker-cli

执行cp命令的两个入口分别位于

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cmd.go#L20

func NewContainerCommand(dockerCli command.Cli) *cobra.Command { ... cmd.AddCommand( ... NewCopyCommand(dockerCli), ...

https://github.com/docker/cli/blob/v20.10.6/cli/command/commands/commands.go#L96

func AddCommands(cmd *cobra.Command, dockerCli command.Cli) { ... cmd.AddCommand( ... hide(container.NewCopyCommand(dockerCli)), ...

两者分别用docker cp命令和docker container cp命令，两者都调用同一个函数NewCopyCommand。

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cp.go#L43

func NewCopyCommand(dockerCli command.Cli) *cobra.Command { var opts copyOptions
 cmd := &cobra.Command{ ... Args: cli.ExactArgs(2), RunE: func(cmd *cobra.Command, args []string) error { if args[0] == "" { return errors.New("source can not be empty") } if args[1] == "" { return errors.New("destination can not be empty") } opts.source = args[0] opts.destination = args[1] return runCopy(dockerCli, opts) }, }
 flags := cmd.Flags() flags.BoolVarP(&opts.followLink, "follow-link", "L", false, "Always follow symbol link in SRC_PATH") flags.BoolVarP(&opts.copyUIDGID, "archive", "a", false, "Archive mode (copy all uid/gid information)") return cmd}

该函数定义了cp命令的参数, 具体实现调用runCopy函数。仅支持从容器复制到宿主机、从宿主机复制到容器，分别调用copyFromContainer，copyToContainer，不支持容器间复制，也不支持仅在宿主机间复制。

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cp.go#L77

func runCopy(dockerCli command.Cli, opts copyOptions) error { srcContainer, srcPath := splitCpArg(opts.source) destContainer, destPath := splitCpArg(opts.destination)
 ...
 switch direction { case fromContainer: return copyFromContainer(ctx, dockerCli, copyConfig) case toContainer: return copyToContainer(ctx, dockerCli, copyConfig) case acrossContainers: return errors.New("copying between containers is not supported") default: return errors.New("must specify at least one container source")

copyFromContainer和copyToContainer主要负责拼装参数,并调用docker engine api

3.1 copyFromContainer

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cp.go#L137-L154

if copyConfig.followLink { srcStat, err := client.ContainerStatPath(ctx, copyConfig.container, srcPath)
 // If the destination is a symbolic link, we should follow it. if err == nil && srcStat.Mode&os.ModeSymlink != 0 { linkTarget := srcStat.LinkTarget ... srcPath = linkTarget }
}

实际复制的操作是由CopyFromContainer函数通过调用api实现的。

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cp.go#L156

content, stat, err := client.CopyFromContainer(ctx, copyConfig.container, srcPath)

下面将调用api得到的输出复制到dstPath。

如果dstPath是-, 就会直接输出到stdout。 

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cp.go#L162-L179

if dstPath == "-" { _, err = io.Copy(dockerCli.Out(), content) return err}

如果之前使用了followLink选项，因为更改了srcPath, 所以要把打包文件中的路径修正回原始的路径

preArchive := contentif len(srcInfo.RebaseName) != 0 { _, srcBase := archive.SplitPathDirEntry(srcInfo.Path) preArchive = archive.RebaseArchiveEntries(content, srcBase, srcInfo.RebaseName)}

最后将打包文件复制到目的路径。

return archive.CopyTo(preArchive, srcInfo, dstPath)

这一过程虽然调用的是docker engine的pkg代码，但完全由cli执行, 主要就是将archive解压到dstPath。

https://github.com/moby/moby/blob/v20.10.6/pkg/archive/copy.go#L402

3.2 copyToContainer

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cp.go#L200-L219

dstStat, err := client.ContainerStatPath(ctx, copyConfig.container, dstPath)
// If the destination is a symbolic link, we should evaluate it.if err == nil && dstStat.Mode&os.ModeSymlink != 0 { linkTarget := dstStat.LinkTarget if !system.IsAbs(linkTarget) { // Join with the parent directory. dstParent, _ := archive.SplitPathDirEntry(dstPath) linkTarget = filepath.Join(dstParent, linkTarget) }
 dstInfo.Path = linkTarget...}

如果srcPath是-, 则目的路径必须是目录(根据帮助文档，设计上就是这样从stdin解压到目录，但不清楚为什么这么设计)

https://github.com/docker/cli/blob/v20.10.6/cli/command/container/cp.go#L236-L242

if srcPath == "-" { content = os.Stdin resolvedDstPath = dstInfo.Path if !dstInfo.IsDir { return errors.Errorf("destination \"%s:%s\" must be a directory", copyConfig.container, dstPath) }}

// Prepare source copy info.srcInfo, err := archive.CopyInfoSourcePath(srcPath, copyConfig.followLink)if err != nil { return err}
srcArchive, err := archive.TarResource(srcInfo)if err != nil { return err}defer srcArchive.Close()dstDir, preparedArchive, err := archive.PrepareArchiveCopy(srcArchive, srcInfo, dstInfo)if err != nil { return err}defer preparedArchive.Close()
resolvedDstPath = dstDircontent = preparedArchive

最后复制到容器，实际复制的操作是由CopyToContainer函数通过调用api实现的。

options := types.CopyToContainerOptions{ AllowOverwriteDirWithFile: false, CopyUIDGID: copyConfig.copyUIDGID,}return client.CopyToContainer(ctx, copyConfig.container, resolvedDstPath, content, options)

4. docker engine api

在cli中调用的api有3个:

• ContainerStatPath

• CopyFromContainer

• CopyToContainer

对应的服务端代码分别为：

• head /containers/{name:.*}/archive headContainersArchive

• get /containers/{name:.*}/archive getContainersArchive

• put /containers/{name:.*}/archive putContainersArchive

下面分析这三个函数的具体流程

4.1 headContainersArchive

(ContainerStatPath)

https://github.com/moby/moby/blob/v20.10.6/api/server/router/container/copy.go#L78

这个api有两个参数，分别是requestUri中传入的name(=ContainerID)和query中传入的path。

解析完name, path参数后，由ContainerStatPath函数处理

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L75

func (daemon *Daemon) ContainerStatPath(name string, path string) (stat *types.ContainerPathStat, err error) { container, err := daemon.GetContainer(name) ... stat, err = daemon.containerStatPath(container, path) ...}

先挂载容器的文件系统和各个存储卷, 然后得到path在rootfs下的resolvedPath和absPath, 再根据它们得到ContainerPathStat

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L152

func (daemon *Daemon) containerStatPath(container *container.Container, path string) (stat *types.ContainerPathStat, err error) { ... if err = daemon.Mount(container); err != nil { return nil, err } ... err = daemon.mountVolumes(container) ... resolvedPath, absPath, err := container.ResolvePath(path) ... return container.StatPath(resolvedPath, absPath)}

ContainerPathStat是这样得到的

https://github.com/moby/moby/blob/v20.10.6/container/archive.go#L51

func (container *Container) StatPath(resolvedPath, absPath string) (stat *types.ContainerPathStat, err error) { ... lstat, err := driver.Lstat(resolvedPath) if err != nil { return nil, err }
 var linkTarget string if lstat.Mode()&os.ModeSymlink != 0 { // Fully evaluate the symlink in the scope of the container rootfs. hostPath, err := container.GetResourcePath(absPath) if err != nil { return nil, err }
 linkTarget, err = driver.Rel(driver.Path(), hostPath) if err != nil { return nil, err }
 // Make it an absolute path. linkTarget = driver.Join(string(driver.Separator()), linkTarget) }
 return &types.ContainerPathStat{ Name: driver.Base(absPath), Size: lstat.Size(), Mode: lstat.Mode(), Mtime: lstat.ModTime(), LinkTarget: linkTarget, }, nil}

最后,将ContainerPathStat返回到header头中 

https://github.com/moby/moby/blob/v20.10.6/api/server/router/container/copy.go#L64

func setContainerPathStatHeader(stat *types.ContainerPathStat, header http.Header) error { statJSON, err := json.Marshal(stat) if err != nil { return err }
 header.Set( "X-Docker-Container-Path-Stat", base64.StdEncoding.EncodeToString(statJSON), )
 return nil}

4.2 getContainersArchive

(CopyFromContainer)

这个api有两个参数，分别是requestUri中传入的name(=ContainerID)和query中传入的path(=srcPath)。

打包完后将数据压缩并返回在body中。

https://github.com/moby/moby/blob/v20.10.6/api/server/router/container/copy.go#L115

func (s *containerRouter) getContainersArchive(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error { v, err := httputils.ArchiveFormValues(r, vars) if err != nil { return err }
 tarArchive, stat, err := s.backend.ContainerArchivePath(v.Name, v.Path) if err != nil { return err } defer tarArchive.Close()
 if err := setContainerPathStatHeader(stat, w.Header()); err != nil { return err }
 w.Header().Set("Content-Type", "application/x-tar") return writeCompressedResponse(w, r, tarArchive)}

调用daemon.containerArchivePath函数打包 

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L100

func (daemon *Daemon) ContainerArchivePath(name string, path string) (content io.ReadCloser, stat *types.ContainerPathStat, err error) { ctr, err := daemon.GetContainer(name) ... content, stat, err = daemon.containerArchivePath(ctr, path) ...

containerArchivePath函数会创建一个关于path的归档文件，并返回一个ContainerPathStat

其中获取ContainerPathStat的过程与headContainersArchive几乎一致 

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L182-L218

func (daemon *Daemon) containerArchivePath(container *container.Container, path string) (content io.ReadCloser, stat *types.ContainerPathStat, err error) { ... if err = daemon.Mount(container); err != nil { return nil, nil, err } ... if err = daemon.mountVolumes(container); err != nil { return nil, nil, err }
 // Normalize path before sending to rootfs path = container.BaseFS.FromSlash(path)
 resolvedPath, absPath, err := container.ResolvePath(path) ... stat, err = container.StatPath(resolvedPath, absPath) ...}

关键在于打包的过程:

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L231-L263

driver := container.BaseFS...opts := archive.TarResourceRebaseOpts(sourceBase, driver.Base(absPath))
data, err := archivePath(driver, sourceDir, opts, container.BaseFS.Path())...content = ioutils.NewReadCloserWrapper(data, func() error { err := data.Close() container.DetachAndUnmount(daemon.LogVolumeEvent) daemon.Unmount(container) container.Unlock() return err})

如果driver(container.BaseFS)有实现ArchivePath函数，则优先使用该函数。

目前localfs均未实现该函数，仅发现remotefs(lcowfs)有此实现 https://github.com/moby/moby/blob/v20.10.6/daemon/graphdriver/lcow/remotefs.go#L83

否则，使用chrootarchive.Tar打包

func archivePath(i interface{}, src string, opts *archive.TarOptions, root string) (io.ReadCloser, error) { if ap, ok := i.(archiver); ok { return ap.ArchivePath(src, opts) } return chrootarchive.Tar(src, opts, root)}

4.2.1 chrootarchive.Tar

https://github.com/moby/moby/blob/v20.10.6/pkg/chrootarchive/archive.go#L101

func Tar(srcPath string, options *archive.TarOptions, root string) (io.ReadCloser, error) { if options == nil { options = &archive.TarOptions{} } return invokePack(srcPath, options, root)}

调用docker-tar命令打包 

https://github.com/moby/moby/blob/v20.10.6/pkg/chrootarchive/archive_unix.go#L178

func invokePack(srcPath string, options *archive.TarOptions, root string) (io.ReadCloser, error) { if root == "" { return nil, errors.New("root path must not be empty") }
 ...
 cmd := reexec.Command("docker-tar", relSrc, root)

docker-tar的实现对应tar函数 

https://github.com/moby/moby/blob/v20.10.6/pkg/chrootarchive/init_unix.go#L17

reexec.Register("docker-tar", tar)

具体实现调用archive.TarWithOptions(关于archive模块的源码分析，暂不在本文展开)， 但在此之前先进行了chroot

func tar() { ... if err := realChroot(root); err != nil { fatal(err) } ...  rdr, err := archive.TarWithOptions(src, &options) if err != nil { fatal(err) } defer rdr.Close()
 if _, err := io.Copy(os.Stdout, rdr); err != nil { fatal(err) }
 os.Exit(0)}

4.2.2 lcowfs

lcowfs通过执行remotefs archivepath path命令打包

https://github.com/moby/moby/blob/v20.10.6/daemon/graphdriver/lcow/remotefs.go#L83

if err := l.runRemoteFSProcess(tarBuf, w, remotefs.ArchivePathCmd, src); err != nil {...

https://github.com/moby/moby/blob/v20.10.6/daemon/graphdriver/lcow/remotefs.go#L127-L129

func (l *lcowfs) runRemoteFSProcess(stdin io.Reader, stdout io.Writer, args ...string) error { ... cmd := fmt.Sprintf("%s %s", remotefs.RemotefsCmd, strings.Join(args, " ")) stderr := &bytes.Buffer{} if err := l.currentSVM.runProcess(cmd, stdin, stdout, stderr); err != nil { return err } ...}

remotefs是一个大命令,archivepath是其中一个子命令。archivePath对应的实现为

https://github.com/moby/moby/blob/v20.10.6/vendor/github.com/Microsoft/opengcs/service/gcsutils/remotefs/remotefs.go#L48

// Commands provide a string -> remotefs function mapping.// This is useful for commandline programs that will receive a string// as the function to execute.var Commands = map[string]Func{ ... ArchivePathCmd: ArchivePath,}

archivepath具体实现如下，打包函数实际为archive.TarWithOptions, 关于archive模块的源码分析，暂不在本文展开。

https://github.com/moby/moby/blob/v20.10.6/vendor/github.com/Microsoft/opengcs/service/gcsutils/remotefs/remotefs.go#L559

// ArchivePath archives the given directory and writes it to out.// Args:// - in = size of json | json of archive.TarOptions// - args[0] = source directory name// Out:// - out = tar file of the archivefunc ArchivePath(in io.Reader, out io.Writer, args []string) error { if len(args) < 1 { return ErrInvalid }
 opts, err := ReadTarOptions(in) if err != nil { return err }
 r, err := archive.TarWithOptions(args[0], opts) if err != nil { return err }
 if _, err := io.Copy(out, r); err != nil { return err } return nil}

4.3 putContainersArchive

(CopyToContainer)

putContainersArchive有4个由客户端传入的参数，分别是requestUri中传入的name(=ContainerID),query中传入的path(=dstPath), noOverwriteDirNonDir(=true), copyUIDGID(=true) 

https://github.com/moby/moby/blob/v20.10.6/client/container_copy.go#L33-L59

query.Set("path", filepath.ToSlash(dstPath)) // Normalize the paths used in the API.// Do not allow for an existing directory to be overwritten by a non-directory and vice versa.if !options.AllowOverwriteDirWithFile { query.Set("noOverwriteDirNonDir", "true")}
if options.CopyUIDGID { query.Set("copyUIDGID", "true")}
apiPath := "/containers/" + containerID + "/archive"

解析完参数后，由putContainersArchive函数解包 

https://github.com/moby/moby/blob/v20.10.6/api/server/router/container/copy.go#L135

func (s *containerRouter) putContainersArchive(ctx context.Context, w http.ResponseWriter, r *http.Request, vars map[string]string) error { ... return s.backend.ContainerExtractToDir(v.Name, v.Path, copyUIDGID, noOverwriteDirNonDir, r.Body)}

containerExtractToDir函数中，一开始仍然是先执行mountrootfs和volume操作 

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L273-L289

func (daemon *Daemon) containerExtractToDir(container *container.Container, path string, copyUIDGID, noOverwriteDirNonDir bool, content io.Reader) (err error) { container.Lock() defer container.Unlock()
 if err = daemon.Mount(container); err != nil { return err } defer daemon.Unmount(container)
 err = daemon.mountVolumes(container) defer container.DetachAndUnmount(daemon.LogVolumeEvent) if err != nil { return err }
 // Normalize path before sending to rootfs' path = container.BaseFS.FromSlash(path) driver := container.BaseFS

获取软链接目的路径，判断目的路径是否是目录，如果不是目录，返回报错

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L315-L318

func (daemon *Daemon) containerExtractToDir(container *container.Container, path string, copyUIDGID, noOverwriteDirNonDir bool, content io.Reader) (err error) { ...
 resolvedPath, err := container.GetResourcePath(absPath) if err != nil { return err }
 stat, err := driver.Lstat(resolvedPath) if err != nil { return err }
 if !stat.IsDir() { return ErrExtractPointNotDirectory }

限制目的路径不能是只读的

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L356-L363

func (daemon *Daemon) containerExtractToDir(container *container.Container, path string, copyUIDGID, noOverwriteDirNonDir bool, content io.Reader) (err error) { ... toVolume, err := checkIfPathIsInAVolume(container, absPath) ... if !toVolume && container.HostConfig.ReadonlyRootfs { return ErrRootFSReadOnly }

调用extractArchive函数解包，如果配置了copyUIDGID，会把文件的uid,gid修改为容器内的主用户，否则保留原文件的uid,gid。 

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L365-L379

func (daemon *Daemon) containerExtractToDir(container *container.Container, path string, copyUIDGID, noOverwriteDirNonDir bool, content io.Reader) (err error) { ... options := daemon.defaultTarCopyOptions(noOverwriteDirNonDir)
 if copyUIDGID { var err error // tarCopyOptions will appropriately pull in the right uid/gid for the // user/group and will set the options. options, err = daemon.tarCopyOptions(container, noOverwriteDirNonDir) if err != nil { return err } }
 if err := extractArchive(driver, content, resolvedPath, options, container.BaseFS.Path()); err != nil { return err }

如果driver(container.BaseFS)有ExtractArchive方法，则使用该方法(目前只有lcowfs实现了该方法)。否则调用chrootarchive.UntarWithRoot

https://github.com/moby/moby/blob/v20.10.6/daemon/archive.go#L34

func extractArchive(i interface{}, src io.Reader, dst string, opts *archive.TarOptions, root string) error { if ea, ok := i.(extractor); ok { return ea.ExtractArchive(src, dst, opts) }
 return chrootarchive.UntarWithRoot(src, dst, opts, root)}

4.3.1 chrootarchive.UntarWithRoot

https://github.com/moby/moby/blob/v20.10.6/pkg/chrootarchive/archive.go#L54

func UntarWithRoot(tarArchive io.Reader, dest string, options *archive.TarOptions, root string) error { return untarHandler(tarArchive, dest, options, true, root)}

如果容器内不存在目的目录，就创建并chown，然后调用invokeUnpack函数。 

https://github.com/moby/moby/blob/v20.10.6/pkg/chrootarchive/archive.go#L66

func untarHandler(tarArchive io.Reader, dest string, options *archive.TarOptions, decompress bool, root string) error { ... if _, err := os.Stat(dest); os.IsNotExist(err) { if err := idtools.MkdirAllAndChownNew(dest, 0755, rootIDs); err != nil { return err } } ... return invokeUnpack(r, dest, options, root) }

调用docker-untar命令解包

https://github.com/moby/moby/blob/v20.10.6/pkg/chrootarchive/archive_unix.go#L61

func invokeUnpack(decompressedArchive io.Reader, dest string, options *archive.TarOptions, root string) error { ... cmd := reexec.Command("docker-untar", dest, root) cmd.Stdin = decompressedArchive ... if err := cmd.Start(); err != nil { w.Close() return fmt.Errorf("Untar error on re-exec cmd: %v", err) } ...}

docker-untar实现对应chrootarchive.untar函数，先chroot,再调用archive.Unpack解包。关于archive模块的源码分析，暂不在本文展开。

https://github.com/moby/moby/blob/v20.10.6/pkg/chrootarchive/archive_unix.go#L25

func untar() { ... if err := chroot(root); err != nil { fatal(err) }
 if err := archive.Unpack(os.Stdin, dst, &options); err != nil { fatal(err) } ...}

4.3.2 lcowfs

lcowfs中调用extractarchive实现解包

https://github.com/moby/moby/blob/v20.10.6/daemon/graphdriver/lcow/remotefs.go#L77

func (l *lcowfs) ExtractArchive(src io.Reader, dst string, opts *archive.TarOptions) error { ... if err := l.runRemoteFSProcess(input, nil, remotefs.ExtractArchiveCmd, dst); err != nil {}

extractarchive命令对应ExtractArchive函数

https://github.com/moby/moby/blob/v20.10.6/vendor/github.com/Microsoft/opengcs/service/gcsutils/remotefs/remotefs.go#L47

var Commands = map[string]Func{ ... ExtractArchiveCmd: ExtractArchive,

ExtractArchive函数中调用archive.Untar

func ExtractArchive(in io.Reader, out io.Writer, args []string) error { ... if err := archive.Untar(in, args[0], opts); err != nil {}