原创大数据基础之Ambari通过Ambari部署Airflow

Posted 2021-02-04 Thinking in BigData

ambari2.7.3（hdp3.1） 安装 airflow1.10

ambari的hdp中原生不支持airflow安装，下面介绍如何通过mpack方式使ambari支持airflow安装：

1 下载

# wget https://github.com/miho120/ambari-airflow-mpack/raw/master/airflow-service-mpack.tar.gz

2 安装

# ambari-server install-mpack --mpack airflow-service-mpack.tar.gz --verbose

3 重启

# ambari-server restart

4 ambari页面没有出现ambari，检查mpack.json

/var/lib/ambari-server/resources/mpacks/airflow-ambari-mpack-1.5.4/mpack.json

      "service_versions_map": [

        {

          "service_name" : "AIRFLOW",

          "service_version" : "1.10.0",

          "applicable_stacks" : [

            {

              "stack_name" : "HDP",

              "stack_version" : "2.4"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "2.5"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "2.6"

            },

            {

              "stack_name" : "HDP",

              "stack_version" : "3.0"

            }

          ]

        }

      ]

需要增加HDP3.1，修改完重新打包

4 卸载

ambari-server uninstall-mpack --mpack-name=airflow-ambari-mpack

5 安装（同上）

6 重启（同上）

打开ambari页面

 

启动worker时可能失败，查看启动命令

2019-01-18 19:09:48,159 - Execute[\'service airflow-worker start\'] {}

查看日志

journalctl -xe

发现有报错

Jan 18 19:04:05 cdp-test-server-01.bj python[15246]: detected unhandled Python exception in \'/var/lib/ambari-agent/cache/common-services/AIRFLOW/1.10.0/package/scripts/airflow_worker_control.py\'
Jan 18 19:04:05 cdp-test-server-01.bj abrt-server[15254]: Executable \'/var/lib/ambari-agent/cache/common-services/AIRFLOW/1.10.0/package/scripts/airflow_worker_control.py\' doesn\'t belong to any package and ProcessUnpackaged is set t
Jan 18 19:04:05 cdp-test-server-01.bj abrt-server[15254]: \'post-create\' on \'/var/spool/abrt/Python-2019-01-18-19:04:05-15246\' exited with 1
Jan 18 19:04:05 cdp-test-server-01.bj abrt-server[15254]: Deleting problem directory \'/var/spool/abrt/Python-2019-01-18-19:04:05-15246\'

可以通过修改配置解决

# grep ProcessUnpackaged /etc/abrt/abrt-action-save-package-data.conf
no

改为yes

 

 

 

ambari具体使用详见：https://www.cnblogs.com/barneywill/p/10268501.html

 

参考：

https://github.com/miho120/ambari-airflow-mpack