CentOS7 上搭建 CDH（6.3.0）

Posted 2022-11-11 jhxxb

这里以四台节点搭建

IP	HostName	OS
192.168.8.5	h5（主）	CentOS 7.5
192.168.8.6	h6（从）	CentOS 7.5
192.168.8.7	h7（从）	CentOS 7.5
192.168.8.8	h8（从）	CentOS 7.5

 

一、节点基本配置

1.yum 源，这里用阿里的（全部节点）

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
sed -i -e ‘/mirrors.cloud.aliyuncs.com/d‘ -e ‘/mirrors.aliyuncs.com/d‘ /etc/yum.repos.d/CentOS-Base.repo
yum makecache

 

2.编辑 hosts 文件（全部节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/configure_network_names.html

vim /etc/hosts

192.168.8.5 h5
192.168.8.6 h6
192.168.8.7 h7
192.168.8.8 h8

 

3.关闭防火墙（全部节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_disable_iptables.html

systemctl disable firewalld 
systemctl stop firewalld

 

4.关闭 SELinux（全部节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_disable_selinux.html

vim /etc/selinux/config

SELINUX=disabled

# sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g‘ /etc/selinux/config && setenforce 0

 

5.安装 NTP 服务（全部节点，其中一台为服务器端，其余为客户端）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cdh_enable_ntp.html

这里以 h5 为 ntp 服务器，其它节点向 h5 同步时间。

具体搭建参考：https://www.cnblogs.com/jhxxb/p/10579816.html

 

6.安装 Python2.7（使用 HUE 的节点，不使用 HUE 不用安装）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_python_27.html

CentOS7 自带的就是 Python2.7，不用安装

yum install python27
source /opt/rh/python27/enable
python --version

 

二、安装 Cloudera Manager 和 CDH

1.安装 CDH 的 yum 源（全部节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/configure_cm_repo.html

curl -o /etc/yum.repos.d/cloudera-manager.repo https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/cloudera-manager.repo
rpm --import https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPM-GPG-KEY-cloudera

 

2.安装 JDK（全部节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_ig_jdk_installation.html

必须安装在 /usr/java/ 目录中，这里使用 CDH 源中带的 JDK

yum install oracle-j2sdk1.8

yum 下载安装可能会很慢，可以下载后本地安装

https://archive.cloudera.com/cm6/6.3.0/redhat7/yum/RPMS/x86_64/

yum install /home/oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm -y

 

3.安装 Cloudera Manager Server（全部节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_cm_server.html

主节点

yum install cloudera-manager-daemons cloudera-manager-server -y

从节点（在集群部署的时候会自动安装，但下载很慢，可以先提前本地安装）

yum install cloudera-manager-daemons cloudera-manager-agent -y

同样 yum 下载安装可能会很慢，可以下载后本地安装

 

4.安装数据库，这里选用 mysql（主节点，也可以放在其它节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html

MySQL 数据库安装参考：https://www.cnblogs.com/jhxxb/p/11156902.html

安装完成后先修改配置

vim /etc/my.cnf

my.cnf 文件

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space.
#Replace ‘/var/lib/mysql/mysql_binary_log‘ with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1

binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

View Code

然后再启动，新建 CDH 所需要的库：https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html#concept_dsg_3mq_bl

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

-- 也可以直接用 root 用户
GRANT ALL ON scm.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;
GRANT ALL ON amon.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;
GRANT ALL ON rman.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;
GRANT ALL ON hue.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;
GRANT ALL ON metastore.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;
GRANT ALL ON sentry.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;
GRANT ALL ON nav.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;
GRANT ALL ON navms.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;
GRANT ALL ON oozie.* TO ‘cdh6‘@‘%‘ IDENTIFIED BY ‘123456‘;

最后安装 MySQL 驱动

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/cm_ig_mysql.html#cmig_topic_5_5_3

yum install wget -y
# 下载
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
# 解压
tar -zxf mysql-connector-java-5.1.46.tar.gz
# 放到指定目录
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

# 删除
rm -rf  mysql-connector-java-5.1.46 mysql-connector-java-5.1.46.tar.gz

 

5.设置 Cloudera Manager 数据库（主节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/prepare_cm_database.html

格式：/opt/cloudera/cm/schema/scm_prepare_database.sh <databaseType>  <databaseName>  <databaseUser>

当 MySQL 或 MariaDB 与 Cloudera Manager Server 位于同一节点时：/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm

当 MySQL 或 MariaDB 在其它节点上时：/opt/cloudera/cm/schema/scm_prepare_database.sh mysql -h db01.example.com --scm-host cm01.example.com scm scm

 

6.安装 CDH 和其他软件（主节点）

https://www.cloudera.com/documentation/enterprise/6/6.3/topics/install_software_cm_wizard.html

systemctl start cloudera-scm-server
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

# 看到如下日志就可以访问 WEB 端了：http://<server_host>:7180，用户名和密码默认都是 admin
INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

子节点启动

systemctl start cloudera-scm-agent

 

---

http://wangshubing.com/2019/01/24/CentOS7.5,CDH6%E5%AE%89%E8%A3%85%E9%83%A8%E7%BD%B2/

https://mp.weixin.qq.com/s/c7c0wOHb7wPr6Kbg3MTU1A