将我的服务部署到 azure kubernetes 服务时，总是出现错误“部署 'xxx' 超出其进度期限”

Posted 2023-02-19

【中文标题】将我的服务部署到 azure kubernetes 服务时，总是出现错误“部署 \'xxx\' 超出其进度期限”

【英文标题】：Always get error "deployment 'xxx' exceeded its progress deadline" when deploy my service to azure kubernetes service将我的服务部署到 azure kubernetes 服务时，总是出现错误“部署 'xxx' 超出其进度期限”

【发布时间】：2020-06-20 09:09:40

【问题描述】：

我是 AKS 的新手，我正在使用 AKS 设置 CI/CD。 我创建了一个带有两个节点池的 azure kubernetes 服务，一个是默认的（nodepool1，带有 linux OS），另一个是我要部署到的 windows os 节点池（wdpool）。

我的 kubernetes 节点池： My azure kubernetes node pools snapshot

我看到我的仓库镜像推送成功了： 'xap' repository is pushed successfully snapshot

kubeevents 报错如下，该如何解决？

Error: failed to start container "xap": Error response from daemon: hcsshim::CreateComputeSystem xap: The container operating system does not match the host operating system.
"

我的部署.yml

apiVersion : apps/v1beta1
kind: Deployment
metadata:
  name: xap 
  namespace: kube-public
spec:
  progressDeadlineSeconds: 900

  replicas: 1
  template:
    metadata:
      labels:
        app: xap 
    spec:
      nodeSelector:
        agentpool: wdpool
        beta.kubernetes.io/os: windows
      containers:
        - name: xap 
          image: xapcontainerregistry.azurecr.io/xap

          ports:
          - containerPort: 80

我的 service.yml

apiVersion: v1
kind: Service
metadata:
    name: xap
spec:
    type: LoadBalancer
    ports:
    - port: 80
    selector:
        app: xap

我在 build yml 中的部署任务：

# Deploy to Kubernetes - Review app with Azure DevSpaces
# Build and push image to Azure Container Registry; Deploy to Azure Kuberentes Services
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- users/jasca/xapaks

resources:
- repo: self

variables:

  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: '49c3dcce-5bc7-4ea4-90fe-4bb701a497e8'
  imageRepository: 'xap'
  containerRegistry: 'xapcontainerregistry.azurecr.io'
  dockerfilePath: '**/Dockerfile'
  tag: '$(Build.BuildId)'
  imagePullSecret: 'xapcontainerregistry4671248d-auth'

  # Agent VM image name
  vmImageName: 'windows-latest'

  # Azure Service connection
  azureConnection: '4e998ed5-bbcb-4495-b19e-1ad6ca32bbe1'

  # Name of the new namespace being created to deploy the PR changes.
  k8sNamespaceForPR: '$(system.pullRequest.sourceBranch)'
  buildConfiguration: 'Debug'

stages:
- stage: Build
  displayName: Build WebContainerHost
  jobs:  
  - job: Build
    displayName: Build
    pool:
#      vmImage: $(vmImageName)
       name: XAP_AKS_Pool
#      name: XapOuterloopProductionOnly
#      demands:
#        - Agent.ComputerName -equals xapagent4
    steps:

    # ignore build tasks, succeed.

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build

  jobs:
  - deployment: Deploy
    condition: and(succeeded(), not(startsWith(variables['Build.SourceBranch'], 'refs/pull/')))
    displayName: Deploy
    pool:
      vmImage: $(vmImageName)
    environment: 'xapplatform.kube-public'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)

          - task: KubernetesManifest@0
            displayName: Deploy to Kubernetes cluster
            inputs:
              rolloutStatusTimeout: 1800

              action: deploy 
              manifests: |
                $(Pipeline.Workspace)/manifests/deployment.yml
                $(Pipeline.Workspace)/manifests/service.yml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository):$(tag)
            timeoutInMinutes: 40



  - deployment: DeployPullRequest
    displayName: Deploy Pull request
    condition: and(succeeded(), startsWith(variables['Build.SourceBranch'], 'refs/pull/'))
    pool:
      vmImage: $(vmImageName)

    environment: 'xapplatform.$(k8sNamespaceForPR)'
    strategy:
      runOnce:
        deploy:
          steps:
          - reviewApp: kube-public

          - task: Kubernetes@1
            displayName: 'Create a new namespace for the pull request'
            inputs:
              command: apply
              useConfigurationFile: true
              inline: ' "kind": "Namespace", "apiVersion": "v1", "metadata":  "name": "$(k8sNamespaceForPR)" '

          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              namespace: $(k8sNamespaceForPR)
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)

          - task: KubernetesManifest@0
            displayName: Deploy to the new namespace in the Kubernetes cluster
            inputs:
              action: deploy
              namespace: $(k8sNamespaceForPR)
              manifests: |
                $(Pipeline.Workspace)/manifests/deployment.yml
                $(Pipeline.Workspace)/manifests/service.yml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository):$(tag)

          - task: Kubernetes@1
            name: get
            displayName: 'Get services in the new namespace'
            continueOnError: true
            inputs:
              command: get
              namespace: $(k8sNamespaceForPR)
              arguments: svc
              outputFormat: jsonpath='http://.items[0].status.loadBalancer.ingress[0].ip:.items[0].spec.ports[0].port'

          # Getting the IP of the deployed service and writing it to a variable for posing comment
          - script: |
              url="$(get.KubectlOutput)"
              message="Your review app has been deployed"
              if [ ! -z "$url" -a "$url" != "http://:" ] 
              then
                message="$message and is available at $url.<br><br>[Learn More](https://aka.ms/testwithreviewapps) about how to test and provide feedback for the app."
              fi
              echo "##vso[task.setvariable variable=GITHUB_COMMENT]$message"

我的 createPullSecret 任务成功：

   ##[section]Starting: Create imagePullSecret
==============================================================================
Task         : Deploy to Kubernetes
Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version      : 0.165.4
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
==============================================================================
            Kubectl Client Version: v1.17.1
            Kubectl Server Version: v1.14.8
==============================================================================
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe delete secret xapcontainerregistry4671248d-auth --namespace kube-public
secret "xapcontainerregistry4671248d-auth" deleted
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe create secret docker-registry xapcontainerregistry4671248d-auth --docker-username *** --docker-password *** --docker-server *** --docker-email ServicePrincipal@AzureRM --namespace kube-public
secret/xapcontainerregistry4671248d-auth created
##[section]Finishing: Create imagePullSecret

我的部署到 Kubernetes 集群任务日志：

##[section]Starting: Deploy to Kubernetes cluster
==============================================================================
Task         : Deploy to Kubernetes
Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version      : 0.165.4
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
==============================================================================
            Kubectl Client Version: v1.17.1
            Kubectl Server Version: v1.14.8
==============================================================================
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe apply -f d:\a\_temp\Deployment_xap_1583635691677,d:\a\_temp\Service_xap_1583635691677 --namespace kube-public
deployment.apps/xap configured
service/xap unchanged
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe rollout status Deployment/xap --timeout 1800s --namespace kube-public
error: deployment "xap" exceeded its progress deadline
Waiting for deployment "xap" rollout to finish: 1 old replicas are pending termination...
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "xap" rollout to finish: 1 old replicas are pending termination...
##[error]Error: error: deployment "xap" exceeded its progress deadline
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe describe Deployment xap --namespace kube-public
Name:                   xap
Namespace:              kube-public
CreationTimestamp:      Fri, 06 Mar 2020 14:16:37 +0000
Labels:                 app=xap
Annotations:            deployment.kubernetes.io/revision: 15
                        kubectl.kubernetes.io/last-applied-configuration:
                          "apiVersion":"apps/v1beta1","kind":"Deployment","metadata":"annotations":,"name":"xap","namespace":"kube-public","spec":"progressDea...
Selector:               app=xap
Replicas:               1 desired | 1 updated | 2 total | 0 available | 2 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  app=xap
  Containers:
   xap:
    Image:        ***/xap
    Port:         80/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  xap-5b95494dd7 (1/1 replicas created)
NewReplicaSet:   xap-79cc65ddf9 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  36m   deployment-controller  Scaled down replica set xap-668784656f to 0
  Normal  ScalingReplicaSet  36m   deployment-controller  Scaled up replica set xap-694857dcd8 to 1
  Normal  ScalingReplicaSet  36m   deployment-controller  Scaled down replica set xap-67fd46c7b5 to 0
  Normal  ScalingReplicaSet  36m   deployment-controller  Scaled up replica set xap-79cc65ddf9 to 1
  Normal  ScalingReplicaSet  10m   deployment-controller  Scaled down replica set xap-694857dcd8 to 0
  Normal  ScalingReplicaSet  10m   deployment-controller  Scaled up replica set xap-5b95494dd7 to 1
For more information, go to https://msasg.visualstudio.com/Bing_and_IPG/_environments/75/providers/kubernetes/12
[command]C:\ProgramData\Chocolatey\bin\kubectl.exe get service/xap -o json --namespace kube-public

    "apiVersion": "v1",
    "kind": "Service",
    "metadata": 
        "annotations": 
            "kubectl.kubernetes.io/last-applied-configuration": "\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":\"annotations\":,\"name\":\"xap\",\"namespace\":\"kube-public\",\"spec\":\"ports\":[\"port\":80],\"selector\":\"app\":\"xap\",\"type\":\"LoadBalancer\"\n"
        ,
        "creationTimestamp": "2020-03-06T14:16:38Z",
        "name": "xap",
        "namespace": "kube-public",
        "resourceVersion": "1743607",
        "selfLink": "/api/v1/namespaces/kube-public/services/xap",
        "uid": "17d862ed-5fb5-11ea-ba44-82c49be4b313"
    ,
    "spec": 
        "clusterIP": "10.0.76.15",
        "externalTrafficPolicy": "Cluster",
        "ports": [
            
                "nodePort": 30742,
                "port": 80,
                "protocol": "TCP",
                "targetPort": 80
            
        ],
        "selector": 
            "app": "xap"
        ,
        "sessionAffinity": "None",
        "type": "LoadBalancer"
    ,
    "status": 
        "loadBalancer": 
            "ingress": [
                
                    "ip": "52.139.245.51"
                
            ]
        
    

service xap external IP is 52.139.245.51
##[error]Rollout status check failed.
##[section]Finishing: Deploy to Kubernetes cluster

【问题讨论】：

提供容器的日志？

如何查看容器的日志？我的容器注册表？

这是一个多容器 pod 吗？ kubectl get pods 会给你 pod name..之后 kubectl 记录 podname

不知道是不是multi container pod，怎么查？

查看您的部署，它是单容器 pod

【参考方案1】：

根据@Arghya-Sadhu的建议，我检查了Azure日志中的日志，“KubeEvent”表，它说

failed to start container "xap": Error response from daemon: hcsshim::CreateComputeSystem xap: The container operating system does not match the host operating system. 

上述错误是由于我的docker building agent使用的是windows server 2016，而azure容器系统是windows server 2019，不匹配...我将我的building agent机器操作系统升级到windows server 2019，问题已解决并成功部署到 Kubernetes。