centos7中安装cdh5.15

Posted 2022-09-06

cdh搭建
1.软件准备

软件	版本
os	Centos 7.3(64位)
cm	cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz
cdh	CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel & CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1 & manifest.json
mysql	mysql-community-common-5.7.27-1.el7.x86_64.rpm & mysql-community-libs-5.7.27-1.el7.x86_64.rpm & mysql-community-libs-compat-5.7.27-1.el7.x86_64.rpm & mysql-community-devel-5.7.27-1.el7.x86_64.rpm & mysql-community-client-5.7.27-1.el7.x86_64.rpm & mysql-community-server-5.7.27-1.el7.x86_64.rpm
jdk	jdk-8u221-linux-x64.rpm
jdbc	mysql-connector-java-5.1.48.tar

2.主机配置规划

ip	配置	角色	需执行以下步骤
192.168.16.236		slave1	3,4,5,6,9
192.168.16.237		master	3,4,5,6,7,8,9,10,11
192.168.16.238		slave2	3,4,5,6,9

3.配置host

shell> vi /etc/hosts

在host文件中加入以下内容：
127.0.0.1 localhost
192.168.16.237 master
192.168.16.236 slave1
192.168.16.238 slave2
4.关闭防火墙和selinux安全工具

shell> systemctl stop firewalld.service（关闭防火墙）
shell> systemctl disable firewalld.service（禁止开机启动）
shell> firewall-cmd --state（状态查询）

关闭linux SELINUX安全内核

shell> setenforce 0（临时有效）

修改配置文件使其永久有效

shell> vi /etc/selinux/config

在config文件中找到SELINUX并设置为：
SELINUX=disabled

shell> reboot
shell> sestatus（状态查询）

5.配置ssh免密登陆和ntp时钟同步（参考https://blog.51cto.com/abezoo/2394680 中软件应用部分）
6.安装jdk及配置java环境变量
查看系统是否自带有jdk并将其卸载

shell> rpm -qa |grep java
shell> rpm -e -nodeps 包名

卸载后安装下载的jdk

shell> rpm -ivh jdk-8u221-linux-x64.rpm

配置环境java环境变量

shell> vi /etc/profile

在profile文件末尾加入以下内容：
export JAVA_HOME=/usr/java/jdk1.8.0_221-amd64
export PATH=.:$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
然后激活上面的配置项

shell> source /etc/profile
shell> echo $JAVA_HOME（查看是否配置成功）

7.安装mysql（参考https://blog.51cto.com/abezoo/2402906）
mysql这里需要注意两点：
（1）mysql的密码策略，mysql默认的中级，我们在这里除了要把其改为低级外，还要将密码的长度设置为3，因为我们后面cdh中通用的scm库的密码通常正好是3位

mysql> set global validate_password_policy=LOW;
mysql> set validate_password_number_count=0;
mysql> set validate_password_length=3;
mysql> SHOW VARIABLES LIKE ‘validate_password%‘; （查看密码策略中的各项参数）

(2)给root用户和scm用户赋权

mysql> grant all privileges on *.* to ‘root‘@‘localhost‘ identified by ‘root‘ with grant option;
mysql> grant all privileges on *.* to ‘root‘@‘127.0.0.1‘ identified by ‘root‘ with grant option;
mysql> grant all privileges on *.* to ‘root‘@‘%‘ identified by ‘root‘ with grant option;

mysql> create user scm;
mysql> grant all privileges on *.* to ‘scm‘@‘localhost‘ identified by ‘scm‘ with grant option;
mysql> grant all privileges on *.* to ‘scm‘@‘127.0.0.1‘ identified by ‘scm‘ with grant option;
mysql> grant all privileges on *.* to ‘scm‘@‘%‘ identified by ‘scm‘ with grant option;
mysql> flush privileges;

8.cdh所需数据库创建

mysql> create database hive default charset utf8;
mysql> create database oozie default charset utf8;
mysql> create database hue default charset utf8;
mysql> create database scm default charset utf8;

9.依赖软件安装

shell> yum install -y psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gassapi chkconfig bind-utils perl

10.cm安装
在每台服务器上创建CM使用的用户

shell> useradd --system --home=/opt/cm-5.15.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
shell> mv CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha1 CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha
shell> tar -zxvf cloudera-manager-centos7-cm5.15.0_x86_64.tar.gz -C /opt/
shell> mv CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel CDH-5.15.0-1.cdh5.15.0.p0.21-el7.parcel.sha manifest.json /opt/cloudera/parcel-repo

解压jdbc，并将其移到相应位置

shell> tar -xvf mysql-connector-java-5.1.48.tar
shell> cd mysql-connector-java-5.1.48
shell> cp mysql-connector-java-5.1.48-bin.jar /opt/cm-5.15.0/share/cmf/lib
shell> cp mysql-connector-java-5.1.48-bin.jar /usr/share/java

初始化cm服务器

shell> /opt/cm-5.15.0/share/cmf/schema/scm_prepare_database.sh mysql -hlocalhost -uroot -proot --scm-host localhost scm scm scm

直到出现your scm database is configured correctly!则表示初始化成功，才可接着配置agent

shell> vi /opt/cm-5.15.0/etc/cloudera-scm-agent/config.ini

找到文件中的server_host并设置为：
server_host=master

将cm复制到slave上

shell> scp -r /opt/cm-5.15.0 root@slave1:/opt
shell> scp -r /opt/cm-5.15.0 root@slave2:/opt

启动服务

shell> /opt/cm-5.15.0/etc/init.d/cloudera-scm-server start
shell> /opt/cm-5.15.0/etc/init.d/cloudera-scm-server statue（查看是否成功启动）
shell> /opt/cm-5.15.0/etc/init.d/cloudera-scm-agent start

注意：master、slave1、slave2三台机器都需要启动agent
输入：http://192.168.16.237:7180/ 就可以访问到CM的管理界面用户名密码都是：admin

11.组件安装