在 Google Cloud Build 上使用 Docker Buildkit

Posted 2023-02-16

【中文标题】在 Google Cloud Build 上使用 Docker Buildkit

【英文标题】：Using Docker Buildkit on Google Cloud Build

【发布时间】：2019-12-02 03:27:04

【问题描述】：

我正在尝试在 Google Cloud Build 上将 BuildKit 与 Docker 一起使用，以便最终可以使用 --secret 标志。我使用Build Enhancements for Docker 作为参考。

当我使用以下命令时，它可以在我的笔记本电脑上运行：DOCKER_BUILDKIT=1 docker build -t hello-world:latest .

当我在 Cloud Build 上运行它时，我收到错误“docker.io/docker/dockerfile:experimental not found”。

知道如何让它在 Cloud Build 上运行吗？

这是设置（注意：我还没有使用 --secret 标志）：

Dockerfile：

#syntax=docker/dockerfile:experimental

FROM node:10.15.3-alpine

RUN mkdir -p /usr/src/app && \
    apk add --no-cache tini

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install --production

COPY . .

RUN chown -R node:node .

USER node

EXPOSE 8080

ENTRYPOINT ["/sbin/tini", "--"]

CMD [ "node", "index.js" ]

cloudbuild.yaml：

steps:

  - id: 'Build'
    name: 'gcr.io/cloud-builders/docker'
    args: [
      'build',
      '-t', 'gcr.io/$PROJECT_ID/hello-world:latest',
      '.'
    ]
    env:
      - "DOCKER_BUILDKIT=1"

云构建日志：

starting build "xxxx"

FETCHSOURCE
Fetching storage object: gs://xxxxx
Copying gs://xxxxx...
/ [0 files][ 0.0 B/ 15.3 KiB] 
/ [1 files][ 15.3 KiB/ 15.3 KiB] 
Operation completed over 1 objects/15.3 KiB. 
BUILD
Already have image (with digest): gcr.io/cloud-builders/docker

#2 [internal] load .dockerignore
#2 digest: sha256:3ce0de94c925587ad30afb764af9bef89edeb62eb891b99694aedb086ee53f50
#2 name: "[internal] load .dockerignore"
#2 started: 2019-07-24 03:21:49.153855989 +0000 UTC
#2 completed: 2019-07-24 03:21:49.195969197 +0000 UTC
#2 duration: 42.113208ms
#2 transferring context: 230B done


#1 [internal] load build definition from Dockerfile
#1 digest: sha256:82b0dcd17330313705522448d60a78d4565304d55c86f55b903b18877d612601
#1 name: "[internal] load build definition from Dockerfile"
#1 started: 2019-07-24 03:21:49.150042849 +0000 UTC
#1 completed: 2019-07-24 03:21:49.189628322 +0000 UTC
#1 duration: 39.585473ms
#1 transferring dockerfile: 445B done


#3 resolve image config for docker.io/docker/dockerfile:experimental
#3 digest: sha256:401713457b113a88eb75a6554117f00c1e53f1a15beec44e932157069ae9a9a3
#3 name: "resolve image config for docker.io/docker/dockerfile:experimental"
#3 started: 2019-07-24 03:21:49.210803849 +0000 UTC
#3 completed: 2019-07-24 03:21:49.361743084 +0000 UTC
#3 duration: 150.939235ms
#3 error: "docker.io/docker/dockerfile:experimental not found"

docker.io/docker/dockerfile:experimental not found
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: exit status 1

笔记本电脑 Docker 版本：

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:39 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:13:06 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Cloud Build Docker 版本：

Step #0 - "Version": Client:
Step #0 - "Version": Version: 18.09.7
Step #0 - "Version": API version: 1.39
Step #0 - "Version": Go version: go1.10.8
Step #0 - "Version": Git commit: 2d0083d
Step #0 - "Version": Built: Thu Jun 27 17:56:17 2019
Step #0 - "Version": OS/Arch: linux/amd64
Step #0 - "Version": Experimental: false
Step #0 - "Version": 
Step #0 - "Version": Server: Docker Engine - Community
Step #0 - "Version": Engine:
Step #0 - "Version": Version: 18.09.3
Step #0 - "Version": API version: 1.39 (minimum version 1.12)
Step #0 - "Version": Go version: go1.10.8
Step #0 - "Version": Git commit: 774a1f4
Step #0 - "Version": Built: Thu Feb 28 05:59:55 2019
Step #0 - "Version": OS/Arch: linux/amd64
Step #0 - "Version": Experimental: false

更新：我注意到我使用的是#syntax=docker/dockerfile:experimental，而链接的文章有#syntax=docker/dockerfile:1.0-experimental。使用 1.0-experimental 时出现同样的错误。

【问题讨论】：

Google Cloud example of using secrets in a build

@Paul 感谢您的链接。但是，这种方法并不能解决我试图通过使用 --secret 标志来解决的问题。我正在使用私有包构建 Node 应用程序，并且需要将 NPM_TOKEN 传递给 docker build 命令。在 Docker 文件中使用环境变量和 ARG 将公开令牌。请参阅 alexandraulsh.com/2019/02/24/docker-build-secrets-and-npmrc 了解我正在尝试实施的解决方案。

@Mark 你找到解决方案了吗？

很遗憾，没有。

【参考方案1】：

当“registry-mirrors”选项与buildkit结合使用时似乎出现问题，然后buildkit前端图像无法获取：

https://github.com/moby/moby/issues/39120

在构建之前拉动它们似乎可以解决问题：

  - name: 'gcr.io/cloud-builders/docker'
    args: ['pull', 'docker/dockerfile:experimental']
  - name: 'gcr.io/cloud-builders/docker'
    args: ['pull', 'docker/dockerfile:1.0-experimental']

【讨论】：

这个。谢谢@troels-liebe-bentsen。但是，现在遇到了这个问题：[1/4] FROM docker.io/library/node:12.18.4-alpine resolve docker.io/library/node:12.18.4-alpine 0.1s done ERROR: docker.io/library /node:12.18.4-alpine 未找到

【参考方案2】：

我遇到了类似的问题并设法解决了。将 docker buildkit 与 gcr.io/cloud-builders/docker 一起使用实际上是不可能的，相反，您必须在 docker daemon 中运行一个 docker 并使用 docker-compose 在旁边运行另一个 docker build。

具体来说，您需要一个docker-compose.yml，它具有：

docker（docker daemon 中的docker） 构建映像的 docker 构建步骤（启用 buildkit） 授权 docker 推送到 gcr 的 docker auth 和推送步骤（您需要创建 creds.json w/服务角色 w/gcs 权限，详情见底部） 为了验证和推送到 gcr，需要使用 creds.json 进行 docker login。查看详情：https://cloud.google.com/container-registry/docs/advanced-authentication

# deploy/app/docker-compose.yml
version: '3.7'
services:
  docker:
    image: "docker:18.09.9-dind"
    privileged: true
    volumes:
      - docker-certs-client:/certs/client
      - docker-certs-ca:/certs/ca
    expose:
      - 2376
    environment:
      - DOCKER_TLS_CERTDIR=/certs
    networks:
      - docker-in-docker-network
  docker-build:
    image: docker:18.09.9
    working_dir: /project
    command: build -t 'gcr.io/$PROJECT_ID/<image>:<tag>'
    privileged: true
    depends_on: 
      - docker
    volumes:
      - docker-certs-client:/certs/client:ro
      - ../../:/project
    environment:
      - DOCKER_TLS_CERTDIR=/certs
      - DOCKER_BUILDKIT=1
    networks:
      - docker-in-docker-network
  docker-push:
    image: docker:18.09.9
    working_dir: /project
    entrypoint: /bin/sh -c
    command: 
      - |
        cat creds.json | docker login -u _json_key --password-stdin https://gcr.io
        docker push 'gcr.io/$PROJECT_ID/<image>:<tag>'
    privileged: true
    depends_on: 
      - docker
    volumes:
      - docker-certs-client:/certs/client:ro
      - ../../:/project
    environment:
      - DOCKER_CERT_PATH=/certs/client
      - DOCKER_HOST=tcp://docker:2376
      - DOCKER_TLS_VERIFY=1
    networks:
      - docker-in-docker-network
volumes:
  docker-certs-ca:
  docker-certs-client:
networks:
  docker-in-docker-network:

然后在你的cloud-build.yaml:

你需要先解密一个 creds.json（必须先创建并加密）——详情：https://cloud.google.com/docs/authentication/getting-started （推送步骤将使用密钥授权 docker 登录到 gcr。） 从 docker-compose 以守护程序模式运行 docker 守护程序（因此它不会阻止构建和推送步骤） 运行构建步骤docker-compose 在docker-compose 中运行身份验证和推送步骤。

# cloud-build.yaml
steps:
  # decrypt gcloud json secret
  - name: gcr.io/cloud-builders/gcloud
    args:
    - kms
    - decrypt
    - --ciphertext-file=deploy/app/creds.json.enc
    - --plaintext-file=creds.json
    - --location=global
    - --keyring=<...>
    - --key=<...>
  # run docker daemon
  - name: 'docker/compose:1.24.1'
    args: ['-f', 'deploy/app/docker-in-docker-compose.yml', 'up', '-d', 'docker']
    env:
      - 'PROJECT_ID=$PROJECT_ID'
  # build image
  - name: 'docker/compose:1.24.1'
    args: ['-f', 'deploy/app/docker-in-docker-compose.yml', 'up', 'docker-build']
    env:
      - 'PROJECT_ID=$PROJECT_ID'
  # docker auth and push to gcr
  - name: 'docker/compose:1.24.1'
    args: ['-f', 'deploy/app/docker-in-docker-compose.yml', 'up', 'docker-push']
    env:
      - 'PROJECT_ID=$PROJECT_ID'
timeout: 600s