azkaban的部署

Posted bujunpeng

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了azkaban的部署相关的知识,希望对你有一定的参考价值。

随着平台上的任务越来越多,且部分任务相互依赖,目前使用的crontab定时调度已经无法满足使用需求.急需一个任务依赖调度系统,笔者以前使用过oozie,但是oozie配置过多,而且维护繁琐,属于重量级系统.所以最终我选择使用azkaban.网上关于oozie和azkaban比较的文章很多,有兴趣可以自行百度.接下来说说azkaban的部署.

一.部署模式

  • 单机模式(solo)
  • 单机集群(1个webServer,1个execServer)
  • 集群模式(1个webServer,多个execServer)
    本文选择第三种模式,即一台机器安装webServer服务,多台机器安装execServer.这种模式是使用最广泛的.

二.部署准备

  1. 需要一个mysql服务器,创建azkaban库,azkaban用户密码,并赋予远程连接.
    技术分享图片
    1 mysql> CREATE DATABASE azkaban;
    2 mysql> CREATE USER azkaban@% IDENTIFIED BY azkaban;
    3 mysql> CREATE USER azkaban@localhost IDENTIFIED BY azkaban;
    4 mysql> grant all privileges on azkaban.* to azkaban@% identified by azkaban;
    5 mysql> grant all privileges on azkaban.* to azkaban@localhost identified by azkaban;
    6 mysql> flush privileges;
    View Code
  2. 下载最新azkaban源文件(azkaban-3.47.0.tar.gz)

三.开始部署

  1. 解压下载的文件azkaban-3.47.0.tar.gz 
  2. 进入目录执行下面代码编译源文件
    ./gradlew distTar
  3. 解压编译好的文件,目录如下:
    azkaban-exec-server/build/distributions/
    azkaban-web-server/build/distributions/
    azkaban-solo-server/build/distributions/
    azkaban-db/build/distributions/
  4. 导入建表语句
    mysql -uazkaban -p
    > SOURCE XXX/create-all-sql-0.1.0-SNAPSHOT.sql;
  5. 创建目录/opt/Azkaban-web-server 并将azkaban-web-server/build/distributions/下文件拷贝至此
  6. 将azkaban-solo-server下conf 目录拷贝到/opt/Azkaban-web-server目录下,修改其中azkaban.properties文件
    技术分享图片
     1 # Azkaban Personalization Settings
     2 azkaban.name=nyhadoop
     3 azkaban.label=My Local Azkaban
     4 azkaban.color=#FF3601
     5 azkaban.default.servlet.path=/index
     6 web.resource.dir=web/
     7 default.timezone.id=Asia/Shanghai
     8 # Azkaban UserManager class
     9 user.manager.class=azkaban.user.XmlUserManager
    10 user.manager.xml.file=conf/azkaban-users.xml
    11 # Loader for projects
    12 executor.global.properties=conf/global.properties
    13 azkaban.project.dir=projects
    14 
    15 database.type=mysql
    16 mysql.port=3306
    17 mysql.host=10.46.67.33
    18 mysql.database=azkaban
    19 mysql.user=azkaban
    20 mysql.password=azkaban
    21 mysql.numconnections=100
    22 
    23 # Velocity dev mode
    24 velocity.dev.mode=false
    25 # Azkaban Jetty server properties.
    26 jetty.ssl.port=8443
    27 # jetty.use.ssl=false
    28 jetty.port=8081
    29 jetty.keystore=keystore
    30 jetty.password=123456
    31 jetty.keypassword=123456
    32 jetty.truststore=keystore
    33 jetty.trustpassword=123456
    34 jetty.maxThreads=25
    35 # Azkaban Executor settings
    36 executor.port=12321
    37 # mail settings
    38 mail.sender=
    39 mail.host=
    40 # User facing web server configurations used to construct the user facing server URLs. They are useful when there is a reverse proxy between Azkaban web servers and users.
    41 # enduser -> myazkabanhost:443 -> proxy -> localhost:8081
    42 # when this parameters set then these parameters are used to generate email links. 
    43 # if these parameters are not set then jetty.hostname, and jetty.port(if ssl configured jetty.ssl.port) are used.
    44 # azkaban.webserver.external_hostname=myazkabanhost.com
    45 # azkaban.webserver.external_ssl_port=443
    46 # azkaban.webserver.external_port=8081
    47 job.failure.email=
    48 job.success.email=
    49 lockdown.create.projects=false
    50 cache.directory=cache
    51 # JMX stats
    52 jetty.connector.stats=true
    53 executor.connector.stats=true
    54 # Azkaban plugin settings
    55 azkaban.jobtype.plugin.dir=plugins/jobtypes
    56 azkaban.use.multiple.executors=true
    57 azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
    58 azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
    59 azkaban.executorselector.comparator.Memory=1
    60 azkaban.executorselector.comparator.LastDispatched=1
    61 azkaban.executorselector.comparator.CpuUsage=1
    View Code
  7. 配置jetty ssl
    技术分享图片
    keytool -keystore keystore -alias jetty -genkey -keyalg RSA
    Enter keystore password:  输入密码
    Re-enter new password: 再次输入密码
    What is your first and last name?
      [Unknown]:  直接回车
    What is the name of your organizational unit?
      [Unknown]:  直接回车
    What is the name of your organization?
      [Unknown]:  直接回车
    What is the name of your City or Locality?
      [Unknown]:  直接回车
    What is the name of your State or Province?
      [Unknown]:  直接回车
    What is the two-letter country code for this unit?
      [Unknown]:  直接回车
    Is CN=YY, OU=YY, O=YY, L=shanghai, ST=shanghai, C=CN correct?
      [no]:  y
    View Code
  8. 将生成的keystore文件 移动到/opt/Azkaban-web-server/目录下
  9. 创建目录/opt/Azkaban-exec-server 并将azkaban-exec-server/build/distributions/下文件拷贝至此
  10. 将azkaban-solo-server下conf 目录拷贝到/opt/Azkaban-exec-server目录下,修改其中azkaban.properties文件
    技术分享图片
    # Azkaban Personalization Settings
    default.timezone.id=Asia/Shanghai
    # Loader for projects
    executor.global.properties=conf/global.properties
    azkaban.project.dir=projects
    
    database.type=mysql
    mysql.port=3306
    mysql.host=10.46.67.33
    mysql.database=azkaban
    mysql.user=azkaban
    mysql.password=azkaban
    mysql.numconnections=100
    
    # Azkaban Executor settings
    executor.maxThreads=50
    executor.port=12321
    executor.flow.threads=30
    # JMX stats
    jetty.connector.stats=true
    executor.connector.stats=true
    # Azkaban plugin settings
    azkaban.jobtype.plugin.dir=plugins/jobtypes
    View Code
  11. 分别在/opt/Azkaban-web-server/conf和/opt/Azkaban-exec-server/conf目录下创建log4j.properties文件
    技术分享图片
    log4j.rootLogger=DEBUG,C
    log4j.appender.C=org.apache.log4j.ConsoleAppender
    log4j.appender.C.Target=System.err
    log4j.appender.C.layout=org.apache.log4j.PatternLayout
    log4j.appender.C.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
    View Code
  12. 分别在/opt/Azkaban-web-server/和/opt/Azkaban-exec-server/目录下创建plugins/jobtypes/commonprivate.properties目录和文件
    技术分享图片
    azkaban.native.lib=false
    execute.as.user=false
    View Code

     

四.运行

  1. /opt/Azkaban-exec-server目录下执行
    1 bin/start-exec.sh
    2 curl http://${executorHost}:${executorPort}/executor?action=activate
  2. /opt/Azkaban-web-server目录下执行
    bin/start-web.sh

     

  3. 浏览器地址 https://{ip}:8443查看服务界面

 













以上是关于azkaban的部署的主要内容,如果未能解决你的问题,请参考以下文章

03-azkaban安装部署

Azkaban 3.x 编译及部署

azkaban配置多节点集群模式部署

azkaban的部署以及简单使用

Azkaban

Azkaban