来自共享 Gitlab 运行器的 SSH 停止工作

Posted 2023-03-23

【中文标题】来自共享 Gitlab 运行器的 SSH 停止工作

【英文标题】：SSH from shared Gitlab runner stopped working

【发布时间】：2020-10-11 15:15:05

【问题描述】：

这确实以前工作过！

我在管道 SSH 中的部署步骤到 DO 框并从 docker 注册表中提取代码。如前所述，这在以前有效，这是我当时在.gitlab-ci.yml 中的deploy 步骤，从here 下的Using SSH 获得了很好的灵感：

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - mkdir -p ~/.ssh
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - eval "$(ssh-agent -S)"
    - ssh-add ~/.ssh/id_rsa
    - ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
  script:
    - ssh -vvv gitlab@$DEPLOYMENT_SERVER_IP
      "docker stop $CI_PROJECT_NAME;
      docker rm $CI_PROJECT_NAME;
      docker container prune -f;
      docker rmi $CI_REGISTRY/$CI_PROJECT_PATH;
      docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY;
      docker pull $CI_REGISTRY/$CI_PROJECT_PATH:latest;
      docker run -d -p $HTTP_PORT:$HTTP_PORT --restart=always -m 800m --init --name $CI_PROJECT_NAME --net $NETWORK_NAME --ip $NETWORK_IP $CI_REGISTRY/$CI_PROJECT_PATH:latest;"

一旦我尝试运行deploy 步骤并失败。返回此错误：

...
 $ mkdir -p ~/.ssh
 $ echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
 $ chmod 600 ~/.ssh/id_rsa
 $ eval "$(ssh-agent -s)"
 Agent pid 22
 $ ssh-add ~/.ssh/id_rsa
 Identity added: /root/.ssh/id_rsa (/root/.ssh/id_rsa)
 $ ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 $ ssh gitlab@$DEPLOYMENT_SERVER_IP "docker stop $CI_PROJECT_NAME; docker rm $CI_PROJECT_NAME; docker container prune -f; docker rmi $CI_REGISTRY/$CI_PROJECT_PATH; docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY; docker pull $CI_REGISTRY/$CI_PROJECT_PATH:latest; docker run -d -p $PORT:$PORT --restart always -m 2g --init --name $CI_PROJECT_NAME --net $NETWORK_NAME --ip $NETWORK_IP $CI_REGISTRY/$CI_PROJECT_PATH:latest;"
 ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
Running after_script
00:02
Uploading artifacts for failed job
00:01
 ERROR: Job failed: exit code 255

我最初设置的步骤

在 DO 框上运行 ssh-keygen -t rsa -b 2048（无密码） 在 DO 框上的 authorized_keys 中添加了

public

密钥 将

private

键复制到 CI 变量 DEPLOYMENT_SERVER_PRIVATE_KEY

我知道端口对 SSH 开放，因为我能够从本地计算机通过 SSH 连接到 gitlab 用户。我现在已将我的部署步骤 （基于来自 here、this article 和 this one 的 cmets） 更改为：

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - eval $(ssh-agent -s)
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - cat ~/.ssh/config
    - echo $CI_REGISTRY_USER
    - ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
  script:
    - ssh -vvv gitlab@$DEPLOYMENT_SERVER_IP
      "docker stop $CI_PROJECT_NAME;
      docker rm $CI_PROJECT_NAME;
      docker container prune -f;
      docker rmi $CI_REGISTRY/$CI_PROJECT_PATH;
      docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY;
      docker pull $CI_REGISTRY/$CI_PROJECT_PATH:latest;
      docker run -d -p $HTTP_PORT:$HTTP_PORT --restart=always -m 800m --init --name $CI_PROJECT_NAME --net $NETWORK_NAME --ip $NETWORK_IP $CI_REGISTRY/$CI_PROJECT_PATH:latest;"

还是没用！ ssh 的详细日志吐了出来：

...
 $ which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )
 /usr/bin/ssh-agent
 $ eval $(ssh-agent -s)
 Agent pid 18
 $ echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
 Identity added: (stdin) ((stdin))
 $ mkdir -p ~/.ssh
 $ chmod 700 ~/.ssh
 $ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
 $ cat ~/.ssh/config
 Host *
    StrictHostKeyChecking no
 $ echo $CI_REGISTRY_USER
 gitlab-ci-token
 $ ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.209.184.138:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 # xxx.xxx.xxx.xxx:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
 $ ssh -vvv gitlab@$DEPLOYMENT_SERVER_IP
 OpenSSH_8.3p1, OpenSSL 1.1.1g  21 Apr 2020
 debug1: Reading configuration data /root/.ssh/config
 debug1: /root/.ssh/config line 1: Applying options for *
 debug1: Reading configuration data /etc/ssh/ssh_config
 debug2: resolve_canonicalize: hostname 134.xxx.xxx.xxx is address
 Pseudo-terminal will not be allocated because stdin is not a terminal.
 debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
 debug2: ssh_connect_direct
 debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
 debug1: connect to address xxx.xxx.xxx.xxx port 22: Connection refused
 ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused
 ERROR: Job failed: exit code 255

我还添加了 -T 选项 suggested here 以禁用伪 tty 分配，但所做的只是从日志中删除伪行。

编辑

查看 DO 框 (/var/log/auth.log) 上的日志，我得到了错误：

Jun 22 15:53:37 exchange-apis sshd[16159]: Connection closed by 35.190.162.232 port 49750 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16160]: Connection closed by 35.190.162.232 port 49754 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16162]: Connection closed by 35.190.162.232 port 49752 [preauth]
Jun 22 15:53:38 exchange-apis sshd[16163]: Unable to negotiate with 35.190.162.232 port 49756: no matching host key type found. Their offer: [e-mail] [preauth]
Jun 22 15:53:38 exchange-apis sshd[16161]: Unable to negotiate with 35.190.162.232 port 49758: no matching host key type found. Their offer: [e-mail] [preauth]

谷歌搜索此错误，常见原因似乎是由于 OpenSSH 放弃了对 DSA 密钥的支持。但是，当我生成一个 RSA 密钥对时，不确定为什么这会影响我。无论如何，运行dpkg --list | grep openssh 会吐出：

ii  openssh-client                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) client, for secure access to remote machines
ii  openssh-server                         1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) server, for secure access from remote machines
ii  openssh-sftp-server                    1:7.6p1-4ubuntu0.3                              amd64        secure shell (SSH) sftp server module, for SFTP access from remote machines

&sshd -v吐槽：

OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017

尽管如此，答案仍然有效； here & here 所以我的deploy 舞台现在是：

deploy:
  stage: deploy
  image: docker:stable-dind
  only:
    - master
  services:
    # Specifying the DinD version here as the latest DinD version introduced a timeout bug
    # Highlighted here: https://forum.gitlab.com/t/gitlab-com-ci-stuck-on-docker-build/34401/2
    - docker:19.03.5-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: ""
  environment:
    name: production
  when: manual
  before_script:
    - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )'
    - mkdir -p ~/.ssh
    - echo "$DEPLOYMENT_SERVER_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\tHostkeyAlgorithms +ssh-dss\n\tPubkeyAcceptedKeyTypes +ssh-dss\n\n" > ~/.ssh/config'
    - cat ~/.ssh/config
    - ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
  script:
    - ssh -oHostKeyAlgorithms=+ssh-dss gitlab@$DEPLOYMENT_SERVER_IP ls

仍然没有看到，我在跑步者的输出和 DO 框上的日志中得到了同样的错误。有什么想法吗？

【参考方案1】：

理想情况下，如果您可以登录 DO 框，您将停止 ssh 服务，并启动 /usr/bin/sshd -de，以便在 SSH 守护进程端建立调试会话，并将日志写入 stderr（而不是系统消息）

但如果你不能，至少尝试生成一个没有密码的 rsa 密钥，用于测试。这意味着您不需要 ssh-agent。 并尝试ssh -Tv gitlab@$DEPLOYMENT_SERVER_IP ls 看看那里产生了什么日志。

试试classic PEM format

ssh-keygen -t rsa -P "" -m PEM

在对管道进行更多编辑后，我注意到实际上是这一行导致了问题：ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts

如果它导致~/.ssh/known_hosts 格式错误，尤其是$DEPLOYMENT_SERVER_IP 设置不正确时，可能会出现这种情况。 尝试将echo "DEPLOYMENT_SERVER_IP='$DEPLOYMENT_SERVER_IP'" 和cat ~/.ssh/known_hosts 命令添加到before_script 部分，以了解更多信息。

【讨论】：

在调试模式下，CI 中的输出相同，服务器中的输出相同。我的 SSH 密钥对已经没有密码。请查看我的编辑以了解我在查看日志时发现的更多信息

@wmash 尝试测试ssh-keygen -t rsa -P "" -m PEM，看看旧格式是否效果更好。

我刚刚尝试过，它成功了！你能解释一下为什么会这样吗

@wmash 要么是因为 -b 2048（并不总是被接受），要么是因为 OpenSSH 格式（-m PEM 切换到旧格式：***.com/a/53645530/6309）。我已经相应地编辑了答案。

VonC 在对管道进行了更多编辑后，我注意到实际上是这一行导致了问题：ssh-keyscan -H $DEPLOYMENT_SERVER_IP >> ~/.ssh/known_hosts。当我删除它时，它成功了。你知道为什么会这样吗？如果你这样做，你能把它放在你的答案中吗？我会接受