大数据平台CDH5.14.2 的安装配置
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了大数据平台CDH5.14.2 的安装配置相关的知识，希望对你有一定的参考价值。
大数据平台CDH5.14.2 的安装配置
标签（空格分隔）： 大数据平台构建
- 一：系统环境初始化
- 二：安装CDH5.14.2 平台
- 三：分配主机与分配角色
一： 系统环境初始化
1.1: 系统环境介绍
系统： CentOS7.5X64
cat /etc/hosts
---
172.17.100.11 node-01.flyfish
172.17.100.12 node-02.flyfish
172.17.100.13 node-03.flyfish
172.17.100.14 node-04.flyfish
172.17.100.15 kbs-01.flyfish
172.17.100.16 kbs-02.flyfish
-----
本次 安装只配置其中三台机器： 172.17.100.11到 172.17.100.13 
1.2: 关闭 firewalld，iptables 与 selinux
yum install -y firewall* iptable*
(1) 关闭firewalld
systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state
(2) 关闭iptables
iptables -F
systemctl stop iptables.service
service iptables save
systemctl disable iptables.service
(3) 禁用 selinux
sed -i ‘/SELINUX/s/enforcing/disabled/‘ /etc/selinux/config
setenforce 0
getenforce 0
sestatus
重启机器
reboot
1.3:无密钥登录配置
做root用户无密钥认证
ssh-keygen ---一直敲回车到最后
cat id_rsa.pub >> authorized_keys
将所有的公钥导入authorized_keys 分发到 所有的 机器的.ssh/ 下面
然后测试
1.4 配置jdk1.7环境 （所有机器都配置）
1): 卸载原来的jdk
rpm -e java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64 java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64 copy-jdk-configs-2.2-3.el7.noarch java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-devel-1.7.0.141-2.6.10.5.el7.x86_64 java-1.8.0-openjdk-devel-1.8.0.131-11.b12.el7.x86_64 java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el7_3.x86_64 tzdata-java-2017b-1.el7.noarch python-javapackages-3.4.1-11.el7.noarch javapackages-tools-3.4.1-11.el7.noarch --nodeps
2) 安装 jdk:
rpm -ivh oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
vim /etc/profile
最后增加：
### JDK ####
export JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
---
java -version 
1.5：chrony 时间同步服务器
所有机器 yum install chrony*
1. node-01.flyfish:
vim /etc/chrony.conf 增加
----
注释掉相关默认的server
server ntp1.aliyun.com iburst
allow 172.17.100.0.0/16
service chronyd start
chkconfig chronyd on
2. node-02.flyfish---node-03.flyfish 同步node-01.flyfish
vim /etc/chrony.conf
----
注释掉默认的server
server 172.17.100.11 iburst
----
service chronyd start
chkconfig chronyd on 
1.6: 安装maridb-server 数据库
1）安装mariadb-server
yum install -y mariadb*
2) 启动 mariadb-server
service mariadb start
chkcofig mariadb on
3) 配置数据库密码：
mysql_secure_installation
4） 登录mariadb-server 数据库
mysql -uroot -pflyfish225.com
1.8 关于系统参数配置
vim /etc/sysctl.conf
##内核默认参数
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
##打开文件数参数(20*1024*1024)
fs.file-max= 20971520
##WEB Server参数
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=1200
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_rmem=4096 87380 8388608
net.ipv4.tcp_wmem=4096 87380 8388608
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_max_tw_buckets = 5000
##TCP补充参数
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
##禁用ipv6
net.ipv6.conf.all.disable_ipv6 =1
net.ipv6.conf.default.disable_ipv6 =1
##swap使用率优化
vm.swappiness=0
---
执行以下命令使参数生效:
# sysctl -p
同步到所有机器
1.7 修改文件句柄数
vim /etc/security/limits.conf
---
* soft nofile 65535
* hard nofile 1029345
* soft nproc unlimited
* hard nproc unlimited
* soft memlock unlimited
* hard memlock unlimited
---
二：安装CDH5.14.2 平台
2.1: 安装配置依赖包(全部节点都安装)
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python2.2：安装cloudera-server 与更改配置 （节点全部安装）
rpm -ivh cloudera-manager*
rpm -ivh enterprise-debuginfo-5.14.3-1.cm5143.p0.4.el7.x86_64.rpm
2.3:在mariadb-server 当中创建CM 的库
mysql -uroot -pflyfish225.com
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to ‘cmf‘@‘%‘ identified by ‘cmf_1234‘ with grant option;
mysql> grant all privileges on cmf.* to ‘cmf‘@‘node-01.flyfish‘ identified by ‘cmf_1234‘ with grant option;
mysql> flush privileges;
2.4:安装离线cdh parcel-repo
#cd /root/software
#cp -p CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel* /opt/cloudera/parcel-repo/
#cp -p manifest.json /opt/cloudera/parcel-repo/
#cd /opt/cloudera/parcel-repo
mv CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha1 (去掉最后的1) CDH-5.14.2-1.cdh5.14.2.p0.3-el7.parcel.sha
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/ （所有主机都要执行这条命令）
2.5 配置jdbc的驱动
rpm -ivh mysql-connector-java-8.0.11-1.el7.noarch.rpm
cd /usr/share/java
cp -p mysql-connector-java-8.0.11.jar mysql-connector-java.jar
chmod 777 mysql-connector-java.jar
2.6 配置数据的注册生成表
1）备份db生成表的文件
cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak
2）生成表
/usr/share/cmf/schema/scm_prepare_database.sh mysql cmf cmf cmf_1234
2.7:修改config.ini 文件
vim /etc/cloudera-scm-agent/config.ini
---
server_host=172.17.100.11
---
scp /etc/cloudera-scm-agent/config.ini node-02.flyfish:/etc/cloudera-scm-agent/
scp /etc/cloudera-scm-agent/config.ini node-03.flyfish:/etc/cloudera-scm-agent/
2.8: 启动CM-SERVER
cd /etc/init.d/
./cloudera-scm-server start
查看日志：
cd /var/log/cloudera-scm-server
tail -100f cloudera-scm-server.log
查看有7180 端口启动表示CM管理server启动
netstat -nultp |grep 7180
使用浏览器web打开：
http://node-01.flyfish:7180
启动agent 端：
cd /etc/init.d/
./cloudera-scm-agent start
三： 分配主机与分配角色
所有节点执行：
#echo never > /sys/kernel/mm/transparent_hugepage/defrag
#echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local
在最下面增加：
---
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
---
同步所有机器
mysql 授权的配置
数据库授权：
mysql -uroot -pflyfish225.com
---
创建hive 所需要的库并授权：
mysql >create database cdh_hive default character set latin1;
字符必须为拉丁(latin1),如果uft8可能会报错Specified key was too long; max key length is 767 bytes
mysql >grant all privileges on cdh_hive.* to ‘hive‘@‘%‘ identified by ‘Hive_123‘ with grant option;
mysql >grant all privileges on cdh_hive.* to ‘hive‘@‘node-01.flyfish‘ identified by ‘Hive_123‘ with grant option;
mysql >flush privileges;
创建oozie 所需要的库
mysql >create database cdh_oozie default character set utf8;
mysql >grant all privileges on cdh_oozie.* to ‘oozie‘@‘%‘ identified by ‘Oozie_123‘ with grant option;
mysql >grant all privileges on cdh_oozie.* to ‘oozie‘@‘node-01.flyfish‘ identified by ‘Oozie_123‘ with grant option;
mysql >flush privileges;
创建hue所需要的库：
mysql >create database cdh_hue default character set utf8;
mysql >grant all privileges on cdh_hue.* to ‘hue’@‘%‘ identified by ‘Hue_123‘ with grant option;
mysql >grant all privileges on cdh_hue.* to ‘hue‘@‘node-01.flyfish‘ identified by ‘Hue_123‘ with grant option;
mysql >flush privileges;
创建CM 的 监控端Activity Monitor 的库
Activity Monitor:
mysql >create database cdh_am default character set utf8;
mysql >grant all privileges on cdh_am.* to ‘am’@‘%‘ identified by ‘Am_123‘ with grant option;
mysql >grant all privileges on cdh_am.* to ‘am‘@‘node-01.flyfish‘ identified by ‘Am_123‘ with grant option;
mysql >flush privileges;
创建RM 的 监控端 Reports Manager 的库
Reports Manager
mysql >create database cdh_rm default character set utf8;
mysql >grant all privileges on cdh_rm.* to ‘rm’@‘%‘ identified by ‘Rm_123‘ with grant option;
mysql >grant all privileges on cdh_rm.* to ‘rm‘@‘node-01.flyfish‘ identified by ‘Rm_123‘ with grant option;
mysql >flush privileges;
以上是关于大数据平台CDH5.14.2 的安装配置的主要内容，如果未能解决你的问题，请参考以下文章