1.5-1.6 oozie部署

Posted 2021-11-28 weiyiming007

一、部署

可参考文档：http://archive.cloudera.com/cdh5/cdh/5/oozie-4.0.0-cdh5.3.6/DG_QuickStart.html

1、解压oozie

安装包：

oozie-4.0.0-cdh5.3.6-src.tar.gz

ext-2.2.zip

##解压
[[email protected] oozie]# tar zxf oozie-4.0.0-cdh5.3.6.tar.gz -C /opt/cdh-5.3.6/

##先停止hadoop集群

[[email protected] hadoop-2.5.0-cdh5.3.6]# sbin/stop-dfs.sh

[[email protected] hadoop-2.5.0-cdh5.3.6]# sbin/stop-yarn.sh

[[email protected] hadoop-2.5.0-cdh5.3.6]# sbin/mr-jobhistory-daemon.sh stop historyserver

2、Oozie服务器架构

3、修改core-site.xml

##/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop/core-site.xml       //添加以下内容
  <property>
    <name>hadoop.proxyuser.root.hosts</name>
    <value>hadoop-senior.ibeifeng.com</value>
  </property>
  
  <property>
    <name>hadoop.proxyuser.root.groups</name>
    <value>*</value>
  </property>

##

root   //允许启动oozie的用户、组

*      //允许所有组

4、oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz

##oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz       //此压缩包是解压oozie后出现的
[[email protected] oozie-4.0.0-cdh5.3.6]# pwd
/opt/cdh-5.3.6/oozie-4.0.0-cdh5.3.6

[[email protected] oozie-4.0.0-cdh5.3.6]# ls |grep hadooplibs
oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz


##解压oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz
[[email protected] oozie-4.0.0-cdh5.3.6]# tar zxf oozie-hadooplibs-4.0.0-cdh5.3.6.tar.gz        //直接解压到oozie安装目录中

此时会多出一个oozie-4.0.0-cdh5.3.6目录；
[[email protected] oozie-4.0.0-cdh5.3.6]# ls oozie-4.0.0-cdh5.3.6/hadooplibs/
hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6          //支持yarn，用这个
hadooplib-2.5.0-mr1-cdh5.3.6.oozie-4.0.0-cdh5.3.6        //支持第一代MapReduce

5、创建libext目录，cp jars

##创建libext目录
[[email protected] oozie-4.0.0-cdh5.3.6]# pwd
/opt/cdh-5.3.6/oozie-4.0.0-cdh5.3.6

[[email protected] oozie-4.0.0-cdh5.3.6]# mkdir libext/



##复制jar包
[[email protected] oozie-4.0.0-cdh5.3.6]# cp -r oozie-4.0.0-cdh5.3.6/hadooplibs/hadooplib-2.5.0-cdh5.3.6.oozie-4.0.0-cdh5.3.6/* libext/


##复制ext-2.2.zip
[[email protected] oozie-4.0.0-cdh5.3.6]# cp /opt/softwares/cdh/oozie/ext-2.2.zip libext/


##为oozie创建war文件
[[email protected] oozie-4.0.0-cdh5.3.6]# bin/oozie-setup.sh prepare-war


##启动hdfs
[[email protected] hadoop-2.5.0-cdh5.3.6]# sbin/hadoop-daemon.sh start namenode

[[email protected] hadoop-2.5.0-cdh5.3.6]# sbin/hadoop-daemon.sh start datanode

6、sharelib

##将新的sharelib上传到hdfs
[[email protected] oozie-4.0.0-cdh5.3.6]# bin/oozie-setup.sh sharelib create -fs hdfs://hadoop-senior.ibeifeng.com:8020 -locallib oozie-sharelib-4.0.0-cdh5.3.6-yarn.tar.gz


##各个目录里都是各个组件需要的jar包
[[email protected] hadoop-2.5.0-cdh5.3.6]# bin/hdfs dfs -ls /user/root/share/lib/lib_20190510132826
Found 9 items
drwxr-xr-x   - root supergroup          0 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/distcp
drwxr-xr-x   - root supergroup          0 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/hcatalog
drwxr-xr-x   - root supergroup          0 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/hive
drwxr-xr-x   - root supergroup          0 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/hive2
drwxr-xr-x   - root supergroup          0 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/mapreduce-streaming
drwxr-xr-x   - root supergroup          0 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/oozie
drwxr-xr-x   - root supergroup          0 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/pig
-rw-r--r--   3 root supergroup       1364 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/sharelib.properties
drwxr-xr-x   - root supergroup          0 2019-05-10 13:28 /user/root/share/lib/lib_20190510132826/sqoop

7、创建数据库

##创建
[[email protected] oozie-4.0.0-cdh5.3.6]# bin/ooziedb.sh create -sqlfile oozie.sql -run DB Connection


##复制mysql驱动
[[email protected] oozie-4.0.0-cdh5.3.6]# cp /opt/softwares/mysql-libs/mysql-connector-java-5.1.27/mysql-connector-java-5.1.27-bin.jar lib/

[[email protected] oozie-4.0.0-cdh5.3.6]# ls lib/ |grep mysql
mysql-connector-java-5.1.27-bin.jar

二、启动oozie

##先配置oozie-site.xml     //大概在200多行，配置hadoop的配置文件路径
<property>
        <name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
        <value>*=/opt/cdh-5.3.6/hadoop-2.5.0-cdh5.3.6/etc/hadoop</value>
        <description>
            Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
            the Hadoop service (JobTracker, HDFS). The wildcard ‘*‘ configuration is
            used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
            the relevant Hadoop *-site.xml files. If the path is relative is looked within
            the Oozie configuration directory; though the path can be absolute (i.e. to point
            to Hadoop client conf/ directories in the local filesystem.
        </description>
    </property>


##启动，其实就是一个tomcat，端口11000
bin/oozied.sh start


##查看
[[email protected] oozie]# jps
21516 Jps
19651 DataNode
21419 Bootstrap        //oozie
19542 NameNode

[[email protected] oozie]# netstat -ntlp |grep 11000
tcp        0      0 :::11000                    :::*                        LISTEN      21718/java 


##访问
在web中： http://hadoop-senior.ibeifeng.com:11000/oozie/