大数据平台 CDH5.12.2 安装部署
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了大数据平台 CDH5.12.2 安装部署相关的知识,希望对你有一定的参考价值。
- 一:初始化环境部署配置
- 二: CDH 5.12.2 部署处理
一: 初始化环境部署处理
1.1 环境准备:
Centos7.3x64 所有主机名配置 配置好 所有机器的无密钥登陆
1.2 主机名配置
172.17.100.131 node01.yangyang.com node01
172.17.100.132 node02.yangyang.com node02
172.17.100.133 node03.yangyang.com node03
172.17.100.134 node04.yangyang.com node04
172.17.100.135 node05.yangyang.com node05
172.17.100.136 node06.yangyang.com node06
1.3 无密钥登录配置
做root用户无密钥认证
ssh-keygen ---一直敲回车到最后
cat id_rsa.pub >> authorized_keys
将所有的公钥导入authorized_keys 分发到 所有的 机器的.ssh/ 下面
然后测试
1.4 关闭 firewalld,iptables 与 selinux
yum install -y firewall* iptable*
(1) 关闭firewalld
systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state
(2) 关闭iptables
iptables -F
systemctl stop iptables.service
service iptables save
systemctl disable iptables.service
(3) 禁用 selinux
sed -i ‘/SELINUX/s/enforcing/disabled/‘ /etc/selinux/config
setenforce 0
getenforce 0
sestatus
重启机器
1.4 所有机器更改centos7.3x64的文件句柄数
vim /etc/security/limits.conf
* soft nofile 65535
* hard nofile 1029345
* soft nproc unlimited
* hard nproc unlimited
* soft memlock unlimited
* hard memlock unlimited
1.5 所有机器 配置好jdk环境 同步所有机器
(1) 卸载原有jdk
rpm -e java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64 java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64 copy-jdk-configs-2.2-3.el7.noarch java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-devel-1.7.0.141-2.6.10.5.el7.x86_64 java-1.8.0-openjdk-devel-1.8.0.131-11.b12.el7.x86_64 java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el7_3.x86_64 tzdata-java-2017b-1.el7.noarch python-javapackages-3.4.1-11.el7.noarch javapackages-tools-3.4.1-11.el7.noarch --nodeps
(2) 安装jdk环境
rpm -ivh jdk-8u151-linux-x64.rpm
1.6 生产环境配置chrony 时间同步服务器
所有机器 yum install chrony*
1. node01:
vim /etc/chrony.conf 增加
----
注释掉相关默认的server
server ntp1.aliyun.com iburst
allow 172.17.100.0.0/16
service chronyd start
chkconfig chronyd on
2. node02---node05 同步node01
vim /etc/chrony.conf
----
注释掉默认的server
server 172.17.100.131 iburst
----
service chronyd start
chkconfig chronyd on
1.7 安装mysql5.7数据库主节点上面安装
卸载:mariadb-libs 的包:
----
#rpm -e mariadb-libs mariadb-devel --nodeps
---
安装mysql5.7 所需要依赖包
# yum install -y perl-JSON libaio
解压 mysql 5.7.21 集成安装包
tar -xvf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar
删掉一个包mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm 这个minimal包不然装不了mysql5.7
# rm -rf mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm
# rpm -ivh mysql-community-*
# mysqld --initialize-insecure --user=mysql
注:上面的安装步骤是用忽略安装mysql包安装顺序,系统将自动识别安装顺序
启动mysql5.7
#service mysqld start
# chkconfig mysqld on
注:初次使用mysql的时候,需要更改mysql 的密码,否则查询不了mysql数据库当中的数据。
#mysql -uroot -p
mysql > set password = password(‘Bigdata_1234‘);
mysql> flush privileges;
1.8 修改系统参数
vim /etc/sysctl.conf
##内核默认参数
kernel.sysrq = 0
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
##打开文件数参数(20*1024*1024)
fs.file-max= 20971520
##WEB Server参数
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1
net.ipv4.tcp_fin_timeout=30
net.ipv4.tcp_keepalive_time=1200
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_rmem=4096 87380 8388608
net.ipv4.tcp_wmem=4096 87380 8388608
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_max_tw_buckets = 5000
##TCP补充参数
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.core.wmem_default = 8388608
net.core.rmem_default = 8388608
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.netdev_max_backlog = 262144
net.ipv4.tcp_max_orphans = 3276800
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_mem = 94500000 915000000 927000000
##禁用ipv6
net.ipv6.conf.all.disable_ipv6 =1
net.ipv6.conf.default.disable_ipv6 =1
##swap使用率优化
vm.swappiness=0
---
执行以下命令使参数生效:
# sysctl -p
同步到所有机器
二: CDH5.12.2 安装与配置
2.1 安装CM 所依赖的包:(全部节点都安装)
yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python
2.2 安装cloudera-server 与更改配置 (节点全部安装)
rpm -ivh cloudera-manager-*
rpm -ivh enterprise-debuginfo-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm
在mysql 当中创建CM 的库
# mysql -uroot -pBigdata_1234
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to ‘cmf‘@‘%‘ identified by ‘cmf_1234‘ with grant option;
mysql> grant all privileges on cmf.* to ‘cmf‘@‘node01.yangyang.com‘ identified by ‘cmf_1234‘ with grant option;
修改cm数据库
#cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak
# vim /etc/cloudera-scm-server/db.properties
2.3 安装 离线cdh parcel-repo
#cd /root/software
#cp -ap CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel* /opt/cloudera/parcel-repo/
#cp -p manifest.json /opt/cloudera/parcel-repo/
#cd /opt/cloudera/parcel-repo
mv CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha1 (去掉 这个 1) CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/ (所有主机都要执行这条命令)
2.4 jdbc 配置
拷贝jdbc 到 java的目录
tar -zxvf mysql-connector-java-5.1.44.tar.gz
cd mysql-connector-java-5.1.44
cp mysql-connector-java-5.1.44-bin.jar /usr/share/java/
cd /usr/share/java/
mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar
2.5 修改config.ini 文件
#cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.inibak
#vim /etc/cloudera-scm-agent/config.ini
修改server_host 为主节点的IP 地址
---
server_host=172.17.100.131
---
同步到所有主机节点
scp /etc/cloudera-scm-agent/config.ini node02:/etc/cloudera-scm-agent/
scp /etc/cloudera-scm-agent/config.ini node03:/etc/cloudera-scm-agent/
2.6 启动CM-server与agent-server
node01.yangyang.com 主机
cd /etc/init.d
./cloudera-scm-server start
查看启动日志
查看启动结果:
需要通过查看server的启动日志文件
/var/log/cloudera-scm-server
tail -f cloudera-scm-server.log
看到jetty 表示启动成功 会有 7180 端口启动:
2.7打开web浏览器:
http://172.17.100.131:7180
用户名: admin
密码: admin
启动 所有的agent 节点
cd /etc/init.d/
./cloudera-scm-agent start
一直到分发结束 -- 然后继续
所有节点执行:
#echo never > /sys/kernel/mm/transparent_hugepage/defrag
#echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local
在最下面增加:
---
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
---
同步所有机器
重新运行界面:
配置角色:
数据库授权:
mysql -uroot -pBigdata_1234
---
创建hive 所需要的库并授权:
mysql >create database cdh_hive default character set latin1;
字符必须为拉丁(latin1),如果uft8可能会报错Specified key was too long; max key length is 767 bytes
mysql >grant all privileges on cdh_hive.* to ‘hive‘@‘%‘ identified by ‘Hive_123‘ with grant option;
mysql >grant all privileges on cdh_hive.* to ‘hive‘@‘node01.yangyang.com‘ identified by ‘Hive_123‘ with grant option;
mysql >flush privileges;
创建oozie 所需要的库
mysql >create database cdh_oozie default character set utf8;
mysql >grant all privileges on cdh_oozie.* to ‘oozie‘@‘%‘ identified by ‘Oozie_123‘ with grant option;
mysql >grant all privileges on cdh_oozie.* to ‘oozie‘@‘node01.yangyang.com‘ identified by ‘Oozie_123‘ with grant option;
mysql >flush privileges;
创建hue所需要的库:
mysql >create database cdh_hue default character set utf8;
mysql >grant all privileges on cdh_hue.* to ‘hue’@‘%‘ identified by ‘Hue_123‘ with grant option;
mysql >grant all privileges on cdh_hue.* to ‘hue‘@‘node01.yangyang.com‘ identified by ‘Hue_123‘ with grant option;
mysql >flush privileges;
创建CM 的 监控端Activity Monitor 的库
Activity Monitor:
mysql >create database cdh_am default character set utf8;
mysql >grant all privileges on cdh_am.* to ‘am’@‘%‘ identified by ‘Am_123‘ with grant option;
mysql >grant all privileges on cdh_am.* to ‘am‘@‘node01.yangyang.com‘ identified by ‘Am_123‘ with grant option;
mysql >flush privileges;
以上是关于大数据平台 CDH5.12.2 安装部署的主要内容,如果未能解决你的问题,请参考以下文章
大数据技术栈-Hadoop3.3.4-完全分布式集群搭建部署-centos7(完全超详细-小白注释版)虚拟机安装+平台部署