从零开始搭建CDH6.2操作手册(离线版)

Posted 老赛不易

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了从零开始搭建CDH6.2操作手册(离线版)相关的知识,希望对你有一定的参考价值。

前言:
安装教程以三台服务器为例,主节点master,从节点worker1、worker2。

一、预备阶段

1、集群机器

2、安装包

jdk:
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

mysql:
mysql-5.7.28.zip
mysql-connector-java-5.1.47.tar.gz

cloudera-repos-6.2.0:
cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm

parcel-6.2.0:
CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha
PHOENIX-1.0.jar
PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel
PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel.sha
manifest.json

安装包下载地址:

链接:https://pan.baidu.com/s/1iTUcWvsiCaRAOTGM8__yww
提取码:90go

二、环境安装

1、关闭防火墙(所有节点)

关闭防火墙可以省去很多麻烦:如ntp服务、clouderaManagerWeb访问等。

systemctl stop firewalld.service

2、修改hostname和hosts(所有节点)

修改主机名:

hostnamectl --transient set-hostname master
hostnamectl --static set-hostname master  # --static可以省略

修改ip映射:

vim /etc/hosts

3、SSH免密(所有节点)

在~/下生成.ssh隐藏文件夹及公钥私钥:

ssh-keygen

然后敲(三个回车),就会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥)

将公钥拷贝到要免密登录的目标机器上:

ssh-copy-id master
ssh-copy-id worker1
ssh-copy-id worker2

测试连接:

4、配置NTP服务(所有节点)

(1)下载ntp的安装包:
链接:https://pan.baidu.com/s/15yJV9A9zn2KBvy08A080Kw
提取码:cdjz
(2)安装命令:

rpm -ivh ntp-4.2.6p5-29.el7.centos.2.x86_64.rpm

安装说明:

直接安装ntp-4.2.6p5-29.el7.centos.2.x86_64.rpm如果不提示缺少依赖就不用再安装另外两个了。
如果提示缺少安装相应的rpm包,也是rpm -ivh …
(3)配置:
首先设置ntp服务器(master)
编辑配置文件(vim /etc/ntp.conf),注释默认ntp服务地址,具体修改部分突出显示,如下:

# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1 
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap
restrict 192.168.79.2 mask 255.255.255.0 nomodify notrap #集群所在网段的网关(Gateway),子网掩码(Genmask)

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 127.0.0.1
fudge 127.0.0.1 stratum 10

#broadcast 192.168.1.255 autokey	# broadcast server
#broadcastclient			# broadcast client
#broadcast 224.0.1.1 autokey		# multicast server
#multicastclient 224.0.1.1		# multicast client
#manycastserver 239.255.254.254		# manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography. 
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor

修改完成后保存退出,并重启ntp(systemctl restart ntpd)。

设置ntp客户端(以worker1为例,其余相同,略)

编辑配置文件(vim /etc/ntp.conf),注释掉默认ntp服务,使用我们自己配置的ntp服务器(即 192.168.79.100),具体修改部分突出显示,如下:

# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1 
restrict ::1

restrict 192.168.79.100 nomodify notrap noquery

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server 192.168.79.100
Fudge 192.168.79.100 stratum 10

#broadcast 192.168.1.255 autokey	# broadcast server
#broadcastclient			# broadcast client
#broadcast 224.0.1.1 autokey		# multicast server
#multicastclient 224.0.1.1		# multicast client
#manycastserver 239.255.254.254		# manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography. 
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor

修改完成后保存退出并重启ntp(systemctl restart ntpd)。

(4)查看状态:
1)启动ntp服务

systemctl start ntpd

2)查看ntp服务器有无和上层ntp连通

ntpstat

3)查看ntp服务器与上层ntp的状态

ntpq -p

(5)设置开机自启动:

systemctl enable ntpd.service

查看从ntp服务状态可能会有如下错误
inappropriate address 192.168.79.100 for the fudge command, line ignored
经测试改了fudge的首字母为F就可以修复上述错误,看网上也都是小写的,很奇怪。
几分钟后查看主从ntp主机的时间就同步了,同步较慢稍等即可。

5、安装JDK(所有节点)

检查是否安装过java

rpm -qa | grep java 

如果有,则先卸载

yum remove java* 

安装jdk

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

配置环境变量

vim /etc/profile
#JDK1.8--添加
export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export PATH=$JAVA_HOME/bin:$PATH

刷新环境变量

source /etc/profile

检查是否安装成功

java -version

java version “1.8.0_181”
Java™ SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot™ 64-Bit Server VM (build 25.181-b13, mixed mode)

6、安装MySql(主节点)

创建目录、上传:
mkdir -p /usr/share/java
mv mysql-connector-java-5.1.47.tar.gz /usr/share/java/mysql-connector-java.jar

安装:
1、解压:unzip mysql-5.7.28.zip -d /root
2、执行mysql_auto_install.sh脚本:sh mysql_auto_install.sh

注:由于服务器本身依赖包存在差别,可能会报如下错误:

error: Failed dependencies:
libaio.so.1()(64bit) is needed by mysql-community-server-5.7.28-1.el7.x86_64
libaio.so.1(LIBAIO_0.1)(64bit) is needed by mysql-community-server-5.7.28-1.el7.x86_64
libaio.so.1(LIBAIO_0.4)(64bit) is needed by mysql-community-server-5.7.28-1.el7.x86_64

解决:yum -y install libaio

重新登录(用户名root,密码root),并创建以下数据库:

create database cmserver default charset utf8 collate utf8_general_ci;
grant all on cmserver.* to 'cmserveruser'@'%' identified by 'root';

create database metastore default charset utf8 collate utf8_general_ci;
grant all on metastore.* to 'hiveuser'@'%' identified by 'root';

create database amon default charset utf8 collate utf8_general_ci;
grant all on amon.* to 'amonuser'@'%' identified by 'root';

create database rman default charset utf8 collate utf8_general_ci;
grant all on rman.* to 'rmanuser'@'%' identified by 'root';

7、修改系统参数(所有节点)

sysctl vm.swappiness=10

echo 'vm.swappiness=10' >> /etc/sysctl.conf

echo never > /sys/kernel/mm/transparent_hugepage/defrag

echo never > /sys/kernel/mm/transparent_hugepage/enabled

echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag'  >> /etc/rc.local

echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled'  >> /etc/rc.local

8、安装CM组件(所有节点)

上传所需安装包(注意保持一致)
master:
worker1、worker2:

安装(master 安装三个,worker1和worker2安装两个)

yum localinstall -y cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm 
yum localinstall -y cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
yum localinstall -y cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm

9、CDH6.2.0(主节点)

检查所需要的离线资源包:

全部上传之后,将不同的包移动到对应的目录下:

mv CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel* /opt/cloudera/parcel-repo/
mv PHOENIX-5.0.0-cdh6.2.0.p0.1308267-el7.parcel* /opt/cloudera/parcel-repo/
mv manifest.json /opt/cloudera/parcel-repo/
mv PHOENIX-1.0.jar /opt/cloudera/csd/

修改配置:
master节点
vim /etc/cloudera-scm-server/db.properties

所有节点:
vim /etc/cloudera-scm-agent/config.ini

10、启动CM服务

1)启动server(master):
systemctl start cloudera-scm-server
systemctl status cloudera-scm-server

监控cloudera-scm-server启动是否成功:
#另开一个窗口,查看相关日志。有异常就解决异常

tail -200f /var/log/cloudera-scm-server/cloudera-scm-server.log

#当在 master上 netstat -tunlp | grep 7180 有内容时,说明我们可以访问web页面了

2)启动agent(所有节点):

systemctl start cloudera-scm-agent
systemctl status cloudera-scm-agent

监控cloudera-scm-agent启动是否成功:
#另开一个窗口,查看相关日志。有异常就解决异常

tail -200f /var/log/cloudera-scm-agent/cloudera-scm-agent.log

11、开始安装CDH集群

这里着重展示需要注意的地方!
192.168.79.100:7180
admin/admin





自定义角色分配



还有许多服务自己按需添加安装
第一次安装完成进入,可能很多都没启动成功爆红。根据提示错误信息,修正配置,重启即可!!

以上是关于从零开始搭建CDH6.2操作手册(离线版)的主要内容,如果未能解决你的问题,请参考以下文章

从零开始搭建CDH6.2操作手册(离线版)

CDH5.7Hadoop集群搭建(离线版)

ESP32c3开发环境搭建 IDF V4.4离线版安装使用

ESP32c3开发环境搭建 IDF V4.4离线版安装使用

ESP32c3开发环境搭建 IDF V4.4离线版安装使用

Unity2019.1中文技术手册离线版