尽管更改了progressDeadlineSeconds，但仍会遇到“超过进度期限”

Posted 2023-02-15

技术标签:

【中文标题】尽管更改了progressDeadlineSeconds，但仍会遇到“超过进度期限”【英文标题】：Keep running into "exceeded its progress dead line" despite changing progressDeadlineSeconds 【发布时间】：2020-04-20 04:38:04 【问题描述】：

我是新的 AKS、ACR 和 DevOps 管道，我正在尝试设置 CI/CD 管道。

我有一个包含 AKS 和 ACR 的资源组设置。 AKS 正在使用Standard_B2s，目前只有一个节点，因为我只是在玩。

在提交到 master 时，图像会自动部署到 ACR - 还没有弄清楚如何设置测试 - 但是在部署到 AKS 时，我只是不断收到：

##[error]error: deployment "client-deployment" exceeded its progress deadline

我已将我的 client.yaml 更改为包含大约一个小时的 progressDeadlineSeconds，因为 10、15 和 20 分钟不起作用：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: client-deployment
spec:
  progressDeadlineSeconds: 3600
  replicas: 1
  selector:
    matchLabels:
      component: client
  template:
    metadata:
      labels:
        component: client
    spec:
      containers:
        - name: client
          image: testappcontainers.azurecr.io/testapp-client
          ports:
            - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: client-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: client
  ports:
    - port: 3000
      targetPort: 3000

我刚刚修改了 Pipelines 为我生成的 azure-pipelines.yml，我目前拥有如下：

# Docker
# Build and push an image to Azure Container Registry
# https://docs.microsoft.com/azure/devops/pipelines/languages/docker

trigger:
- master

resources:
- repo: self

variables:
  # Container registry service connection established during pipeline creation
  dockerRegistryServiceConnection: <dockerRegistryServiceConnection_key>
  imageRepository: 'testapp'
  containerRegistry: 'testappcontainers.azurecr.io'
  dockerfilePath: '$(Build.SourcesDirectory)'
  tag: '$(Build.BuildId)'
  imagePullSecret: <imagePullSecret_key>

  # Agent VM image name
  vmImageName: 'ubuntu-latest'

stages:
- stage: Build
  displayName: Build and push stage
  jobs:  
  - job: Build
    displayName: Build
    pool:
      vmImage: $(vmImageName)
    steps:
    - task: Docker@2
      displayName: Build and push client image to container registry
      inputs:
        command: buildAndPush
        repository: $(imageRepository)-client
        dockerfile: $(dockerfilePath)/client/Dockerfile
        containerRegistry: $(dockerRegistryServiceConnection)
        tags: |
          $(tag)

    - upload: manifests
      artifact: manifests

- stage: Deploy
  displayName: Deploy stage
  dependsOn: Build

  jobs:
  - deployment: Deploy
    displayName: Deploy job
    pool:
      vmImage: $(vmImageName)
    environment: 'testapp.default'
    strategy:
      runOnce:
        deploy:
          steps:
          - task: KubernetesManifest@0
            displayName: Create imagePullSecret
            inputs:
              action: createSecret
              secretName: $(imagePullSecret)
              dockerRegistryEndpoint: $(dockerRegistryServiceConnection)

          - task: KubernetesManifest@0
            displayName: Deploy to Kubernetes cluster
            inputs:
              action: deploy
              manifests: |
                $(Pipeline.Workspace)/manifests/client.yaml
              imagePullSecrets: |
                $(imagePullSecret)
              containers: |
                $(containerRegistry)/$(imageRepository):$(tag)

这也是失败任务的日志：

##[debug]Evaluating condition for step: 'Deploy to Kubernetes cluster'
##[debug]Evaluating: SucceededNode()
##[debug]Evaluating SucceededNode:
##[debug]=> True
##[debug]Result: True
##[section]Starting: Deploy to Kubernetes cluster
==============================================================================
Task         : Deploy to Kubernetes
Description  : Use Kubernetes manifest files to deploy to clusters or even bake the manifest files to be used for deployments using Helm charts
Version      : 0.162.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/deploy/kubernetes-manifest
==============================================================================
##[debug]agent.TempDirectory=/home/vsts/work/_temp
##[debug]loading inputs and endpoints
##[debug]loading INPUT_ACTION
##[debug]loading INPUT_KUBERNETESSERVICECONNECTION
##[debug]loading INPUT_STRATEGY
##[debug]loading INPUT_TRAFFICSPLITMETHOD
##[debug]loading INPUT_PERCENTAGE
##[debug]loading INPUT_BASELINEANDCANARYREPLICAS
##[debug]loading INPUT_MANIFESTS
##[debug]loading INPUT_CONTAINERS
##[debug]loading INPUT_IMAGEPULLSECRETS
##[debug]loading INPUT_RENDERTYPE
##[debug]loading INPUT_DOCKERCOMPOSEFILE
##[debug]loading INPUT_HELMCHART
##[debug]loading INPUT_KUSTOMIZATIONPATH
##[debug]loading INPUT_RESOURCETOPATCH
##[debug]loading INPUT_RESOURCEFILETOPATCH
##[debug]loading INPUT_MERGESTRATEGY
##[debug]loading INPUT_SECRETTYPE
##[debug]loading ENDPOINT_AUTH_<token>
##[debug]loading ENDPOINT_AUTH_SCHEME_<token>
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZUREENVIRONMENT
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_AZURETENANTID
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_ROLEBINDINGNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SECRETNAME
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_APITOKEN
##[debug]loading ENDPOINT_AUTH_PARAMETER_<token>_SERVICEACCOUNTCERTIFICATE
##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
##[debug]loading SECRET_CONTAINER_PASSWORD
##[debug]loading SECRET_CONTAINER_USERNAME
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loaded 32
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]Agent.SkipCertValidation=undefined
##[debug]SYSTEM_HOSTTYPE=build
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]Build.BuildNumber=20191231.5
##[debug]Build.DefinitionName=test-app
##[debug]System.DefinitionId=4
##[debug]Agent.JobName=Deploy job
##[debug]System.TeamProject=test-app
##[debug]Build.BuildId=41
##[debug]System.TeamProject=test-app
##[debug]namespace=null
##[debug]containers=***/testapp:41
##[debug]imagePullSecrets=testappcontainers<key>-auth
##[debug]manifests=/home/vsts/work/1/manifests/client.yaml
##[debug]percentage=0
##[debug]strategy=none
##[debug]trafficSplitMethod=pod
##[debug]baselineAndCanaryReplicas=0
##[debug]arguments=null
##[debug]secretArguments=null
##[debug]secretType=dockerRegistry
##[debug]secretName=null
##[debug]dockerRegistryEndpoint=null
##[debug]kubernetesServiceConnection=<token>
##[debug]<token> data namespace = default
##[debug]System.TeamFoundationCollectionUri=https://dev.azure.com/thetestcompany/
##[debug]System.HostType=build
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]Build.SourceBranchName=master
##[debug]Build.Repository.Provider=TfsGit
##[debug]Build.Repository.Uri=https://thetestcompany@dev.azure.com/thetestcompany/test-app/_git/test-app
##[debug]agent.proxyurl=undefined
##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
##[debug]AZURE_HTTP_USER_AGENT=VSTS_<hash>_build_4_0
##[debug]Agent.ProxyUrl=undefined
##[debug]Agent.CAInfo=undefined
##[debug]Agent.ClientCert=undefined
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/node_modules/azure-pipelines-tool-lib/lib.json
##[debug]system.culture=en-US
##[debug]check path : /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]adding resource file: /home/vsts/work/_tasks/KubernetesManifest_<hash>/0.162.1/task.json
##[debug]system.culture=en-US
##[debug]action=deploy
##[debug]kubernetesServiceConnection=<token>
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]<token> data authorizationType = AzureSubscription
##[debug]<token>=https://testappk8s-dns-<key>.hcp.westus.azmk8s.io/
##[debug]<token> auth param serviceAccountCertificate = ***
##[debug]<token> auth param apiToken = ***
##[debug]set KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]<token> data acceptUntrustedCerts = undefined
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which 'kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]System.DefaultWorkingDirectory=/home/vsts/work/1/s
##[debug]defaultRoot: '/home/vsts/work/1/s'
##[debug]findOptions.allowBrokenSymbolicLinks: 'false'
##[debug]findOptions.followSpecifiedSymbolicLink: 'true'
##[debug]findOptions.followSymbolicLinks: 'true'
##[debug]matchOptions.debug: 'false'
##[debug]matchOptions.nobrace: 'true'
##[debug]matchOptions.noglobstar: 'false'
##[debug]matchOptions.dot: 'true'
##[debug]matchOptions.noext: 'false'
##[debug]matchOptions.nocase: 'false'
##[debug]matchOptions.nonull: 'false'
##[debug]matchOptions.matchBase: 'false'
##[debug]matchOptions.nocomment: 'false'
##[debug]matchOptions.nonegate: 'false'
##[debug]matchOptions.flipNegate: 'false'
##[debug]pattern: '/home/vsts/work/1/manifests/client.yaml'
##[debug]findPath: '/home/vsts/work/1/manifests/client.yaml'
##[debug]statOnly: 'true'
##[debug]found 1 paths
##[debug]applying include pattern
##[debug]1 matches
##[debug]1 final results
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]New K8s objects after addin imagePullSecrets are :["apiVersion":"apps/v1","kind":"Deployment","metadata":"name":"client-deployment","spec":"progressDeadlineSeconds":3600,"replicas":1,"selector":"matchLabels":"component":"client","template":"metadata":"labels":"component":"client","spec":"containers":["name":"client","image":"***/testapp-client","ports":["containerPort":3000]],"imagePullSecrets":["name":"testappcontainers1741032e-auth"],"apiVersion":"v1","kind":"Service","metadata":"name":"client-cluster-ip-service","spec":"type":"ClusterIP","selector":"component":"client","ports":["port":3000,"targetPort":3000]]
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]agent.tempDirectory=/home/vsts/work/_temp
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: apply
##[debug]/usr/bin/kubectl arg: ["-f","/home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   apply
##[debug]   -f
##[debug]   /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl apply -f /home/vsts/work/_temp/Deployment_client-deployment_1577816701782,/home/vsts/work/_temp/Service_client-cluster-ip-service_1577816701782 --namespace default
deployment.apps/client-deployment unchanged
service/client-cluster-ip-service unchanged
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: ["rollout","status"]
##[debug]/usr/bin/kubectl arg: Deployment/client-deployment
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   rollout
##[debug]   status
##[debug]   Deployment/client-deployment
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl rollout status Deployment/client-deployment --namespace default
error: deployment "client-deployment" exceeded its progress deadline
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]which '/usr/bin/kubectl'
##[debug]found: '/usr/bin/kubectl'
##[debug]/usr/bin/kubectl arg: get
##[debug]/usr/bin/kubectl arg: service/client-cluster-ip-service
##[debug]/usr/bin/kubectl arg: ["-o","json"]
##[debug]/usr/bin/kubectl arg: ["--namespace","default"]
##[debug]exec tool: /usr/bin/kubectl
##[debug]arguments:
##[debug]   get
##[debug]   service/client-cluster-ip-service
##[debug]   -o
##[debug]   json
##[debug]   --namespace
##[debug]   default
[command]/usr/bin/kubectl get service/client-cluster-ip-service -o json --namespace default

    "apiVersion": "v1",
    "kind": "Service",
    "metadata": 
        "annotations": 
            "kubectl.kubernetes.io/last-applied-configuration": "\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":\"annotations\":,\"name\":\"client-cluster-ip-service\",\"namespace\":\"default\",\"spec\":\"ports\":[\"port\":3000,\"targetPort\":3000],\"selector\":\"component\":\"client\",\"type\":\"ClusterIP\"\n"
        ,
        "creationTimestamp":         "name": "client-cluster-ip-service",
        "namespace": "default",
        "resourceVersion": "1234045",
        "selfLink": "/api/v1/namespaces/default/services/client-cluster-ip-service",
        "uid": "5f077159-2bdd-11ea-af20-3eaa105eb2b3"
    ,
    "spec": 
        "clusterIP": "10.0.181.220",
        "ports": [
            
                "port": 3000,
                "protocol": "TCP",
                "targetPort": 3000
            
        ],
        "selector": 
            "component": "client"
        ,
        "sessionAffinity": "None",
        "type": "ClusterIP"
    ,
    "status": 
        "loadBalancer": 
    

##[debug]KUBECONFIG=/home/vsts/work/_temp/kubectlTask/1577816701759/config
##[debug]set KUBECONFIG=
##[debug]Processed: ##vso[task.setvariable variable=KUBECONFIG;issecret=false;]
##[debug]task result: Failed
##[error]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.issue type=error;]error: deployment "client-deployment" exceeded its progress deadline
##[debug]Processed: ##vso[task.complete result=Failed;]error: deployment "client-deployment" exceeded its progress deadline
##[section]Finishing: Deploy to Kubernetes cluster

然后在 Azure CLI 中，它显示部署在那里，但没有可用的 pod：

eox-dev@Azure:~$ kubectl get deployments
NAME                READY   UP-TO-DATE   AVAILABLE   AGE
client-deployment   0/1     1            0           3h47m
eox-dev@Azure:~$ kubectl describe deployment client-deployment
Name:                   client-deployment
Namespace:              default
CreationTimestamp:      Tue, 31 Dec 2019 15:50:30 +0000
Labels:                 <none>
Annotations:            deployment.kubernetes.io/revision: 1
                        kubectl.kubernetes.io/last-applied-configuration:
                          "apiVersion":"apps/v1","kind":"Deployment","metadata":"annotations":,"name":"client-deployment","namespace":"default","spec":"progre...
Selector:               component=client
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  component=client
  Containers:
   client:
    Image:        testappcontainers.azurecr.io/testapp-client
    Port:         3000/TCP
    Host Port:    0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  <none>
NewReplicaSet:   client-deployment-5688bdc69c (1/1 replicas created)
Events:          <none>

那么我在这里做错了什么？

【问题讨论】：

您部署的容器“client-deployment”无法启动...您能粘贴此容器中的日志吗？好的，这比我一直在看的内容要丰富得多：

Error from server (BadRequest): container "client" in pod "client-deployment-5688bdc69c-hxlcf" is waiting to start: trying and failing to pull image

... 看起来它由于某种原因无法提取图像。至少现在除了一个非描述性错误之外，我还有一些事情要做。 【参考方案1】：

我认为可能导致此问题（尤其是在 AKS 中）的原因是您的集群在部署之前是否对容器运行了运行状况检查或活动性检查。由于 CrashLoopBackOff，您的集群识别出您的应用程序未运行并进入等待状态

只需在“/”上的 API 应用程序上创建一个默认 GET 路由，返回 200

【讨论】：

【参考方案2】：

即使命名空间仍然出现错误，我也遇到了类似的问题 “超过了进度期限正在等待部署“XX-service”部署完成：1 个更新的副本中有 0 个可用...”

由于拉取图片失败

添加 .yaml 中的“imagePullPolicy: Always”修复了我的推出拉问题

【讨论】：

有时在我的项目中删除和部署需要时间来处理和创建新的应该在添加一些时间后完成，并且“部署策略：金丝雀”【参考方案3】：

来自服务器的错误（BadRequest）：容器中的容器“客户端” “client-deployment-5688bdc69c-hxlcf”正在等待开始：尝试和图片拉不出来

根据我的经验，这与imagePullSecrets 和Kubernetes namespace 更为相关。

在您的Create imagePullSecret 和Deploy to Kubernetes cluster 任务中，我看到您没有为任务参数提供值：namespace。这将导致创建一个名为 default 的新命名空间，因为您未指定命名空间。

并且，由createSecret 操作生成的kubernetes secret 为每个命名空间分开。一言以蔽之，不同的命名空间有不同的秘密值：

秘密存储在给定的命名空间中，并且只能被访问通过同一命名空间中的 pod。

现在，让我们回到构建编译过程。

在您的 yml 定义中，Create imagePullSecret 将为新命名空间 default 创建一个 secret，该命名空间由任务自动创建，因为您没有提供给定的命名空间值。

那么，在下一个任务Deploy to Kubernetes cluster，由于同样的原因，这里的任务将重新创建另一个新的命名空间default(注意：这与上一个不同）。此外，您可以从日志中看到此进度：

此时，上一个任务生成的secret 将不适用于当前的命名空间。但是，如您所知，ACR 是一个私有容器注册表，我们的系统必须验证kubernetes secret 是否可用。

此外，在您的 Deploy to Kubernetes 集群任务中，您将存储库指定为 $(imageRepository)，这与您将映像推送到 $(imageRepository)-client 的存储库不同。

这也可以在您的日志中检查：

这就是为什么你的 Kubernetes 中没有可用的节点，并且你也未能拉取镜像。

为避免此问题，请确保您在 KubernetesManifest@0 任务中提供 namespace 值。

      - task: KubernetesManifest@0
        displayName: Create imagePullSecret
        inputs:
          action: createSecret
          secretName: $(imagePullSecret)
          namespace: $(k8sNamespace)
          dockerRegistryEndpoint: $(DRServiceConnection)

      - task: KubernetesManifest@0
        displayName: Deploy to Kubernetes cluster
        inputs:
          action: deploy
          namespace: $(k8sNamespace)
          manifests: |
            $(System.ArtifactsDirectory)/manifests/deployment.yml
          imagePullSecrets: |
            $(imagePullSecret)
          containers: |
            $(containerRegistry)/$(imageRepository)-client:$(tag)

每个命名空间的 imagePullSecrets 的秘密

【讨论】：

以上是关于尽管更改了progressDeadlineSeconds，但仍会遇到“超过进度期限”的主要内容，如果未能解决你的问题，请参考以下文章

尽管通过赋值更改了数组，但 Vue.js 列表没有更新

尽管我使用了 Adapter.notifyDataSetChanged()，但数据集更改后 Android ListView 不刷新

尽管对 Web 配置和主机工厂进行了更改，但仍会出现 WCF 错误

尽管我们在父组件中更改了两次属性的值，但 ngOnChanges 没有被调用两次

尽管属性已更改，应用程序不会立即更新视图

尽管显示从方法块内部更改的值，但从 setter 设置私有值“余额”时出错