利用先电云iaas平台搭建apache官方大数据平台（ambari2.7+hdp3.0）
Posted huangwentao
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了利用先电云iaas平台搭建apache官方大数据平台（ambari2.7+hdp3.0）相关的知识，希望对你有一定的参考价值。
一、ambari架构解析
二、基础环境配置
以两台节点为例来组件Hadoop分布式集群，这里采用的系统版本为Centos7 1511，如下表所示：
|
主机名 |
内存 |
硬盘 |
IP地址 |
角色 |
|
master |
8192MB |
100G |
192.168.200.6 |
Ambari-Server Ambari-Agent |
|
slaver |
8192MB |
100G |
192.168.200.7 |
Ambari-Agent |
搭建所需各软件版本信息
①ambari-2.7.1.0-centos7.tar.gz
②jdk-8u151-linux-x64.tar.gz
③HDP-3.0.1.0-centos7-rpm.tar.gz
④HDP-UTILS-1.1.0.22-centos7.tar
⑤CentOS-7-x86_64-DVD-1511.iso
1.在iaas平台创建两台云主机
- 主机名、主机名解析配置
vi /etc/hosts
- 挂载镜像并配置本地yum源
首先将所需镜像、安装包等，通过secureFX上传至master 的/opt目录
Master节点
3.1进入/opt目录，解压ambari.tar.gz压缩包
[root@master ~]# cd /opt
[root@master opt]# ls
ambari-2.7.1.0-centos7.tar.gz HDP-3.0.1.0-centos7-rpm.tar.gz
centos HDP-UTILS-1.1.0.22-centos7.tar
[root@master opt]# tar -zxvf ambari-2.7.1.0-centos7.tar.gz
解压完成后获得一个ambari的文件夹
[root@master ~]# ls
CentOS-7-x86_64-DVD-1511.iso
[root@master ~]# mount -o loop CentOS-7-x86_64-DVD-1511.iso /mnt/
[root@master ~]# mkdir /opt/centos
[root@master ~]# cp -rvf /mnt/* /opt/centos/
[root@master ~]# cd /etc/yum.repos.d/
[root@master yum.repos.d]# ls
CentOS-Base.repo CentOS-Debuginfo.repo CentOS-Sources.repo
CentOS-CR.repo CentOS-fasttrack.repo CentOS-Vault.repo
[root@master yum.repos.d]# rm -rf *
[root@master yum.repos.d]# vi local.repo
[centos]
name=centos
baseurl=file:///opt/centos
gpgcheck=0
enable=1
[ambari]
name=ambari
baseurl=file:///opt/ambari/centos7/2.7.1.0-169/
gpgcheck=0
enable=1
保存退出
清理源，并列出源
[root@master ~]# yum clean all
Loaded plugins: fastestmirror
Cleaning repos: ambari centos
Cleaning up everything
Cleaning up list of fastest mirrors
[root@master ~]# yum list
[root@master ~]# yum install vsftpd
[root@master ~]# vi /etc/vsftpd/vsftpd.conf
添加
anon_root=/opt/
3.2配置开机启动vsftpd
[root@master ~]# systemctl enable vsftpd
[root@master ~]# systemctl start vsftpd
Slaver节点
[root@slaver ~]# cd /etc/yum.repos.d/
[root@slaver yum.repos.d]# ls
CentOS-Base.repo CentOS-Debuginfo.repo CentOS-Sources.repo
CentOS-CR.repo CentOS-fasttrack.repo CentOS-Vault.repo
[root@slaver yum.repos.d]# rm -rf *
[root@slaver yum.repos.d]# vi ftp.repo
[centos]
name=centos
baseurl=ftp://192.168.200.6/centos
gpgcheck=0
enable=1
[ambari]
name=ambari
baseurl=ftp://192.168.200.6/ambari/centos7/2.7.1.0-169/
gpgcheck=0
enable=1
保存退出
- 配置NTP服务
Master节点
[root@master ~]# yum -y install ntp
[root@master ~]# vi /etc/ntp.conf
4.1将原有server注释
添加
server 127.127.1.0
fudge 127.127.1.0 stratum 10
[root@master ~]# systemctl enable ntpd
[root@master ~]# systemctl start ntpd
Slaver节点
[root@slaver ~]# yum -y install ntpdate
[root@slaver ~]# ntpdate master
[root@slaver ~]# systemctl enable ntpdate
- 配置SSH 免密访问
Master节点
[root@master ~]# yum install openssh-clients
[root@master ~]# ssh-keygen
[root@master ~]# ssh-copy-id slaver.hadoop
Slaver节点
[root@slaver ~]# yum install openssh-clients
[root@slaver ~]# ssh-keygen
[root@slaver ~]# ssh-copy-id master.hadoop
- 禁用Transparent Huge Pages
操作系统后台有一个叫做khugepaged的进程，它会一直扫描所有进程占用的内存，在可能的情况下会把4kpage交换为Huge Pages，在这个过程中，对于操作的内存的各种分配活动都需要各种内存锁，直接影响程序的内存访问性能，并且，这个过程对于应用是透明的，在应用层面不可控制,对于专门为4k page优化的程序来说，可能会造成随机的性能下降现象。
master & slaver
[root@master ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@master ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@master ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always advise [never]
重启后失效，需要再次执行
- 安装配置jdk
Master节点
[root@master ~]# mkdir /usr/jdk64/
[root@master ~]# tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/jdk64/
[root@master ~]# vi /etc/profile
7.1添加以下两条变量
export JAVA_HOME=/usr/jdk64/jdk1.8.0_151
export PATH=$JAVA_HOME/bin:$PATH
[root@master ~]# source /etc/profile
[root@master ~]# java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
Slaver节点
[root@slaver ~]# mkdir /usr/jdk64/
[root@slaver ~]# scp 192.168.200.6://root/jdk-8u151-linux-x64.tar.gz .
jdk-8u151-linux-x64.tar.gz 100% 181MB 90.5MB/s 00:02
[root@slaver ~]# tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/jdk64/
[root@slaver ~]# vi /etc/profile
export JAVA_HOME=/usr/jdk64/jdk1.8.0_151
export PATH=$JAVA_HOME/bin:$PATH
[root@slaver ~]# source /etc/profile
[root@slaver ~]# java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
- 安装httpd服务
8.1安装httpd
[root@master ~]# yum -y install httpd
[root@master ~]# systemctl enable httpd.service
[root@master ~]# systemctl start httpd.service
[root@master ~]# systemctl status httpd.service
8.2解压HDP
将/opt目录下的
HDP-3.0.1.0-centos7-rpm.tar.gz
HDP-UTILS-1.1.0.22-centos7.tar
解压，并复制到 /var/www/html
[root@master opt]# tar -zvxf HDP-3.0.1.0-centos7-rpm.tar.gz -C /var/www/html/
[root@master opt]# tar -zvxf HDP-UTILS-1.1.0.22-centos7.tar -C /var/www/html/
三、ambari-server 安装配置
Master节点
[root@master ~]# yum -y install ambari-server
1.安装MariaDB数据库
[root@master ~]# yum install mariadb mariadb-server mysql-connector-java
1.1启动数据库服务
[root@master ~]# systemctl enable mariadb
[root@master ~]# systemctl start mariadb
1.2配置数据库服务
[root@master ~]# mysql_secure_installation
Set root password? [Y/n] y （设置密码为bigdata）
Remove anonymous users? [Y/n] y
Disallow root login remotely? [Y/n] n
Remove test database and access to it? [Y/n] y
Reload privilege tables now? [Y/n] y
[root@master ~]# mysql -uroot -pbigdata
MariaDB [(none)]> create database ambari;
MariaDB [(none)]> grant all privileges on ambari.* to ‘ambari‘@‘localhost‘ identified by ‘bigdata‘;
MariaDB [(none)]> grant all privileges on ambari.* to ‘ambari‘@‘%‘ identified by ‘bigdata‘;
MariaDB [(none)]> use ambari;
MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
MariaDB [ambari]> quit
- 配置ambari-server
[root@master ~]# vi /etc/profile
添加
export buildNumber=2.7.1.0
[root@master ~]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is ‘disabled‘
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/jdk64/jdk1.8.0_151
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)?
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
ambari-admin-2.7.1.0.169.jar
....
Ambari repo file doesn‘t contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server ‘setup‘ completed successfully.
[root@master ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
- 启动ambari-server服务
[root@master ~]# ambari-server start
DB configs consistency check: no errors and warnings were found.
Ambari Server ‘start‘ completed successfully.
四、ambari-agent 安装配置
master & slaver
1.安装ambari-agent
[root@master ~]# yum -y install ambari-agent
[root@slaver ~]# yum -y install ambari-agent
- 修改变量
[root@master ~]# vi /etc/ambari-agent/conf/ambari-agent.ini
[root@slaver ~]# vi /etc/ambari-agent/conf/ambari-agent.ini
将[server]项修改为master.hadoop
hostname=master.hadoop
3.重启ambari-agent服务
[root@master ~]# ambari-agent restart
[root@slaver ~]# ambari-agent restart
五、部署Hadoop集群
1.访问WEB界面
打开浏览器输入IP:8080访问ambari的web界面
本机为http://192.168.200.6:8080
默认用户名 admin
默认密码 admin
- 创建新的集群
3.1命名集群
3.2选择本地仓库源
选择redhat 7
http://192.168.200.6/HDP/centos7/3.0.1.0-187/
http://192.168.200.6/HDP-UTILS/centos7/1.1.0.22/
3.3配置主机名
3.4主机状态注册检查
3.5服务安装选择
这里我们选择了
YARN + MapReduce2
HDFS
ZooKeeper
Ambari Metrics
Spark2
3.6服务分配大师
默认即可。
3.7节点服务选择
设置Grafana密码
默认下一步即可
确认部署
开始部署
以上是关于利用先电云iaas平台搭建apache官方大数据平台（ambari2.7+hdp3.0）的主要内容，如果未能解决你的问题，请参考以下文章