Kubernetes Cron-Job闲置通知

Posted 2023-01-31

我有一个cronjob，它创建postgres备份作业。我想通过webhook将通知发送到具有cronjob状态失败或成功的松弛通道。如何添加条件或指定Job的状态并发送到Slack？我想下面的curl请求也可以使用，但是如果发现任何错误，请发出警告。

kind: CronJob
metadata:
  name: standup
spec:
  schedule: "* 17 * * 1-5"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: standup
            image: busybox
            resources:
              requests:
                cpu: 1m
                memory: 100Mi
            env:
              - args: /bin/sh
              - -c 
              - curl -X POST -H 'Content-type: application/json' --data '"text":"Hello, World!"' https://hooks.slack.com/services/TQPCENFHP/
      restartPolicy: OnFailure 

---

~ semural$ kubectl logs $pods -n database
The following backups are available in specified backup path:
Added `s3` successfully.
[2020-04-13 14:24:46 UTC]      0B postgresql-cluster/


NAME                                SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
postgresql-postgresql-helm-backup   0 0 * * *   False     0        8h              18h

NAME                                           COMPLETIONS   DURATION   AGE
postgresql-postgresql-helm-backup-1586822400   1/1           37s        8h
postgresql-postgresql-helm-backup-list         1/1           2s         18h
postgresql-postgresql-helm-pgmon               1/1           49s        18h

答案

我认为我们可以创建一个简单的脚本来获取cronjob的状态：

import json
import os
from kubernetes import client, config, utils
from kubernetes.client.rest import ApiException
from api.exceptions import BatchApiNamespaceNotExistedException

class Constants:
    BACKOFF_LIMIT = 1
    STATUS_RUNNING = "RUNNING"
    STATUS_SUCCEED = "SUCCEED"
    STATUS_FAILED = "FAILED"
    STATUS_NOT_FOUND = "NOT FOUND"

class KubernetesApi:
    def __init__(self):
        try:
            config.load_incluster_config()
        except:
            config.load_kube_config()
        self.configuration = client.Configuration()
        self.api_instance = client.BatchV1Api(client.ApiClient(self.configuration))
        self.api_instance_v1_beta = client.BatchV1beta1Api(client.ApiClient(self.configuration))

    def get_job_status(self, job):
        if job is not None:
            total_failed_pod = job.status.failed or 0
            total_succeeded_pod = job.status.succeeded or 0
            if total_failed_pod + total_succeeded_pod < Constants.BACKOFF_LIMIT:
                return Constants.STATUS_RUNNING
            elif total_succeeded_pod > 0:
                return Constants.STATUS_SUCCEED
            return Constants.STATUS_FAILED
        return Constants.STATUS_NOT_FOUND

    def get_cron_job_status(self, namespace):
        try:
            cron_job_list = self.api_instance_v1_beta.list_namespaced_cron_job(namespace=namespace,
                                                                          watch=False)
        except ApiException as e:
            raise BatchApiNamespaceNotExistedException("Exception when calling BatchV1Api->list_namespaced_cron_job: %s\n" % e)

        for cron_job in cron_job_list.items:
          if cron_job.status.active is not None:
            for active_cron_job in cron_job.status.active:
              job = self.api_instance.read_namespaced_job(namespace=namespace,
                                                       name=active_cron_job.name)
              if job_status == Constants.STATUS_FAILED:
                # Do whatever you want in there
                print(job_status)

因此，如果状态失败，那么我们可以将日志发送到松弛状态。