azkaban的部署
Posted bujunpeng
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了azkaban的部署相关的知识,希望对你有一定的参考价值。
随着平台上的任务越来越多,且部分任务相互依赖,目前使用的crontab定时调度已经无法满足使用需求.急需一个任务依赖调度系统,笔者以前使用过oozie,但是oozie配置过多,而且维护繁琐,属于重量级系统.所以最终我选择使用azkaban.网上关于oozie和azkaban比较的文章很多,有兴趣可以自行百度.接下来说说azkaban的部署.
一.部署模式
- 单机模式(solo)
- 单机集群(1个webServer,1个execServer)
- 集群模式(1个webServer,多个execServer)
本文选择第三种模式,即一台机器安装webServer服务,多台机器安装execServer.这种模式是使用最广泛的.
二.部署准备
- 需要一个mysql服务器,创建azkaban库,azkaban用户密码,并赋予远程连接.
1 mysql> CREATE DATABASE azkaban; 2 mysql> CREATE USER ‘azkaban‘@‘%‘ IDENTIFIED BY ‘azkaban‘; 3 mysql> CREATE USER ‘azkaban‘@‘localhost‘ IDENTIFIED BY ‘azkaban‘; 4 mysql> grant all privileges on azkaban.* to ‘azkaban‘@‘%‘ identified by ‘azkaban‘; 5 mysql> grant all privileges on azkaban.* to ‘azkaban‘@‘localhost‘ identified by ‘azkaban‘; 6 mysql> flush privileges;
- 下载最新azkaban源文件(azkaban-3.47.0.tar.gz)
三.开始部署
- 解压下载的文件azkaban-3.47.0.tar.gz
- 进入目录执行下面代码编译源文件
./gradlew distTar
- 解压编译好的文件,目录如下:
azkaban-exec-server/build/distributions/ azkaban-web-server/build/distributions/ azkaban-solo-server/build/distributions/ azkaban-db/build/distributions/
- 导入建表语句
mysql -uazkaban -p > SOURCE XXX/create-all-sql-0.1.0-SNAPSHOT.sql;
- 创建目录/opt/Azkaban-web-server 并将azkaban-web-server/build/distributions/下文件拷贝至此
- 将azkaban-solo-server下conf 目录拷贝到/opt/Azkaban-web-server目录下,修改其中azkaban.properties文件
1 # Azkaban Personalization Settings 2 azkaban.name=nyhadoop 3 azkaban.label=My Local Azkaban 4 azkaban.color=#FF3601 5 azkaban.default.servlet.path=/index 6 web.resource.dir=web/ 7 default.timezone.id=Asia/Shanghai 8 # Azkaban UserManager class 9 user.manager.class=azkaban.user.XmlUserManager 10 user.manager.xml.file=conf/azkaban-users.xml 11 # Loader for projects 12 executor.global.properties=conf/global.properties 13 azkaban.project.dir=projects 14 15 database.type=mysql 16 mysql.port=3306 17 mysql.host=10.46.67.33 18 mysql.database=azkaban 19 mysql.user=azkaban 20 mysql.password=azkaban 21 mysql.numconnections=100 22 23 # Velocity dev mode 24 velocity.dev.mode=false 25 # Azkaban Jetty server properties. 26 jetty.ssl.port=8443 27 # jetty.use.ssl=false 28 jetty.port=8081 29 jetty.keystore=keystore 30 jetty.password=123456 31 jetty.keypassword=123456 32 jetty.truststore=keystore 33 jetty.trustpassword=123456 34 jetty.maxThreads=25 35 # Azkaban Executor settings 36 executor.port=12321 37 # mail settings 38 mail.sender= 39 mail.host= 40 # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users. 41 # enduser -> myazkabanhost:443 -> proxy -> localhost:8081 42 # when this parameters set then these parameters are used to generate email links. 43 # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used. 44 # azkaban.webserver.external_hostname=myazkabanhost.com 45 # azkaban.webserver.external_ssl_port=443 46 # azkaban.webserver.external_port=8081 47 job.failure.email= 48 job.success.email= 49 lockdown.create.projects=false 50 cache.directory=cache 51 # JMX stats 52 jetty.connector.stats=true 53 executor.connector.stats=true 54 # Azkaban plugin settings 55 azkaban.jobtype.plugin.dir=plugins/jobtypes 56 azkaban.use.multiple.executors=true 57 azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus 58 azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1 59 azkaban.executorselector.comparator.Memory=1 60 azkaban.executorselector.comparator.LastDispatched=1 61 azkaban.executorselector.comparator.CpuUsage=1
- 配置jetty ssl
keytool -keystore keystore -alias jetty -genkey -keyalg RSA Enter keystore password: 输入密码 Re-enter new password: 再次输入密码 What is your first and last name? [Unknown]: 直接回车 What is the name of your organizational unit? [Unknown]: 直接回车 What is the name of your organization? [Unknown]: 直接回车 What is the name of your City or Locality? [Unknown]: 直接回车 What is the name of your State or Province? [Unknown]: 直接回车 What is the two-letter country code for this unit? [Unknown]: 直接回车 Is CN=YY, OU=YY, O=YY, L=shanghai, ST=shanghai, C=CN correct? [no]: y
- 将生成的keystore文件 移动到/opt/Azkaban-web-server/目录下
- 创建目录/opt/Azkaban-exec-server 并将azkaban-exec-server/build/distributions/下文件拷贝至此
- 将azkaban-solo-server下conf 目录拷贝到/opt/Azkaban-exec-server目录下,修改其中azkaban.properties文件
# Azkaban Personalization Settings default.timezone.id=Asia/Shanghai # Loader for projects executor.global.properties=conf/global.properties azkaban.project.dir=projects database.type=mysql mysql.port=3306 mysql.host=10.46.67.33 mysql.database=azkaban mysql.user=azkaban mysql.password=azkaban mysql.numconnections=100 # Azkaban Executor settings executor.maxThreads=50 executor.port=12321 executor.flow.threads=30 # JMX stats jetty.connector.stats=true executor.connector.stats=true # Azkaban plugin settings azkaban.jobtype.plugin.dir=plugins/jobtypes
- 分别在/opt/Azkaban-web-server/conf和/opt/Azkaban-exec-server/conf目录下创建log4j.properties文件
log4j.rootLogger=DEBUG,C log4j.appender.C=org.apache.log4j.ConsoleAppender log4j.appender.C.Target=System.err log4j.appender.C.layout=org.apache.log4j.PatternLayout log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
- 分别在/opt/Azkaban-web-server/和/opt/Azkaban-exec-server/目录下创建plugins/jobtypes/commonprivate.properties目录和文件
azkaban.native.lib=false execute.as.user=false
四.运行
- /opt/Azkaban-exec-server目录下执行
1 bin/start-exec.sh 2 curl http://${executorHost}:${executorPort}/executor?action=activate
- /opt/Azkaban-web-server目录下执行
bin/start-web.sh
- 浏览器地址 https://{ip}:8443查看服务界面
以上是关于azkaban的部署的主要内容,如果未能解决你的问题,请参考以下文章