Kubernetes 中部署的私有存储库 Airflow 的 image_pull_secrets 错误

Posted 2023-02-16

【中文标题】Kubernetes 中部署的私有存储库 Airflow 的 image_pull_secrets 错误

【英文标题】：Error in image_pull_secrets for private repository Airflow deployed in Kubernetes

【发布时间】：2021-01-23 21:05:46

【问题描述】：

我已经在 azure kubernetes 中部署了 apache 气流。

apache 气流的 Helm 存储库：https://github.com/apache/airflow/tree/master/chart

AKS 版本：1.16.13

一旦我部署了气流，我就用这个 dag 对其进行了测试：

from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator


default_args = 
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.utcnow(),
    'email': ['[e-mail]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)


dag = DAG(
    'kubernetes_sample', default_args=default_args, schedule_interval=timedelta(minutes=10))


start = DummyOperator(task_id='run_this_first', dag=dag)

passing = KubernetesPodOperator(namespace='default',
                          image="python:3.8-slim-buster",
                          cmds=["python3","-c"],
                          arguments=["print('hello world')"],
                          labels="foo": "bar",
                          name="passing-test",
                          task_id="passing-task",
                          get_logs=True,
                          dag=dag
                          )


passing.set_upstream(start)

它工作正常。现在我想使用我自己的图像。为此，我正在使用 azure 容器，并遵循本指南：https://airflow.readthedocs.io/en/latest/howto/operator/kubernetes.html，我正在使用此代码创建访问我的 azure 注册表的秘密：

kubectl create secret docker-registry testquay \
    --docker-server=quay.io \
    --docker-username=<Profile name> \
    --docker-password=<password>

我构建了我的映像并在本地进行了测试，它正在运行。我将图像上传到 azure 容器注册表并编写以下 dag：

from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
from airflow.contrib.kubernetes import secret
import logging
import os
import sys
import traceback 

try:
    env_var_secret = secret.Secret(
        deploy_type='env',
        deploy_target='VERSION_NUMBER',
        secret='myregistrykey',
        key='VERSION_NUMBER',
            )
    default_args = 
        'owner': 'airflow',
        'depends_on_past': False,
        'start_date': datetime.utcnow(),
        'email': ['[e-mail]'],
        'email_on_failure': False,
        'email_on_retry': False,
        'retries': 1,
        'retry_delay': timedelta(minutes=5)
    

    dag = DAG(
        'jordi_test_get_secert2', default_args=default_args, schedule_interval=timedelta(minutes=10))


    start = DummyOperator(task_id='run_this_first', dag=dag)

    quay_k8s = KubernetesPodOperator(
            namespace='default',
            name="passing-test7",
            image='docker.io/test-pai-1',
            image_pull_secrets=env_var_secret,
            task_id="passing-task6",
            get_logs=True,
            dag=dag
        )


    start >> quay_k8s

except Exception as e:

    error_message = 
            "message": "An internal error ocurred"
            ,"error": str(e)
            , "error information" : str(sys.exc_info())
            , "traceback": str(traceback.format_exc())
        
    logging.info(error_message)

然后给我这个错误：

 File "/home/airflow/.local/lib/python3.6/site-packages/airflow/kubernetes/pod_generator.py", line 272, in __init__
    for image_pull_secret in image_pull_secrets.split(','):
AttributeError: 'Secret' object has no attribute 'split'

遵循本指南：https://airflow.readthedocs.io/en/latest/howto/operator/kubernetes.html

from airflow import DAG
from datetime import datetime, timedelta
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator
from kubernetes.client import models as k8s
import logging
import os
import sys
import traceback 

try:

    default_args = 
        'owner': 'airflow',
        'depends_on_past': False,
        'start_date': datetime.utcnow(),
        'email': ['[e-mail]'],
        'email_on_failure': False,
        'email_on_retry': False,
        'retries': 1,
        'retry_delay': timedelta(minutes=5)
    

    dag = DAG(
        'jordi_test2', default_args=default_args, schedule_interval=timedelta(minutes=10))


    start = DummyOperator(task_id='run_this_first', dag=dag)

    
    quay_k8s = KubernetesPodOperator(
    namespace='default',
    name="passing-test7",
    image='docker.io/test-pai-1',
    image_pull_secrets=[k8s.V1LocalObjectReference('myregistrykey')],
    task_id="passing-task6",
    get_logs=True,
    dag=dag
)



    start >> quay_k8s

except Exception as e:

    error_message = 
            "message": "An internal error ocurred"
            ,"error": str(e)
            , "error information" : str(sys.exc_info())
            , "traceback": str(traceback.format_exc())
        
    logging.info(error_message)

但是给我这个错误：

for image_pull_secret in image_pull_secrets.split(','):
AttributeError: 'list' object has no attribute 'split'

如果我查看 KubernetesPodOperator 的 Airflow 文档：https://airflow.apache.org/docs/stable/_api/airflow/contrib/operators/kubernetes_pod_operator/index.html

说：

 image_pull_secrets (str) – Any image pull secrets to be given to the pod. If more than one secret is required, provide a comma separated list: secret_a,secret_b

正确的写法是怎样的？

【问题讨论】：

您正在给“image_pull_secrets”一个secret.Secret 对象而不是字符串，如源代码中的类型所示。尝试改为"myregistrykey"

【参考方案1】：

会要求澄清作为评论，但我没有声誉。所以提供一个带有一些假设的解决方案。

由于您提供了最新版 Airflow 的文档链接，我假设您正在使用该版本。 您发布的代码未更改。

这里你的秘密被命名为testquay

kubectl create secret docker-registry testquay \
    --docker-server=quay.io \
    --docker-username=<Profile name> \
    --docker-password=<password>

但是，您在发布的代码中将其引用为 myregistrykey，该代码旨在遵循您引用的示例。

image_pull_secrets=[k8s.V1LocalObjectReference('myregistrykey')],

它应该根据示例引用秘密名称。

image_pull_secrets=[k8s.V1LocalObjectReference('testquay')],

对于第一个 DAG 代码，我不相信您可以以这种方式传递 Secrets 对象。这些意味着在运行时作为卷或环境变量注入 k8s pod。 https://github.com/apache/airflow/blob/v1-10-stable/airflow/kubernetes/secret.py#L35-L40