在先电IAAS平台中，搭建先电bigdata平台

Posted 2021-03-22 huangwentao

以两台节点为例来组件Hadoop分布式集群，这里采用的系统版本为Centos7，如下表所示：

主机名	内存	硬盘	IP地址	角色
master	8192MB	100G	192.168.200.131	Ambari-Server Ambari-Agent
slave1	4096MB	100G	192.168.200.133	Ambari-Agent

1.1 配置主机名

# master

# hostnamectl set-hostname master

# hostname

master

# slave1

# hostnamectl set-hostname slave1

# hostname

slave1

1.2 修改hosts文件

# master

# vi /etc/hosts

192.168.200.131 master.hadoop master

192.168.200.133 slave1.hadoop

# slave1

# vi /etc/hosts

192.168.200.131 master.hadoop

192.168.200.133 slave1.hadoop slave1

注意：主机名映射采用FQDN格式。

1.3 修改yum源

# master

将XianDian-BigData-v2.2.iso挂在到/mnt目录下，将其中的ambari解压到/opt目录下，并在master节点配置ftp服务。 

注意：

因为安装大数据相关软件包时，可能会用到相关依赖软件包，所以需要配置Centos7 Yum源，这里可以采用IAAS中的Centos7 Yum源。

# master & slave1

# cd /etc/yum.repos.d/

# rm -vf *

配置Yum源

# vi ambari.repo

[centos7]

baseurl=ftp://192.168.2.10/centos7/

（ 注：具体的yum源根据真实环境配置，本次为实验室测试环境 )

gpgcheck=0

enabled=1

name=centos

[ambari]

name=ambari

baseurl=ftp://10.0.3.61/ambari

（ 注：具体的yum源根据真实环境配置，本次为实验室测试环境 )

enabled=1

gpgcheck=0

 

# master

# yum -y install httpd

将/mnt/文件夹中HDP-2.6.1.0和HDP-UTILS-1.1.0.21两个文件夹拷贝到/var/www/html/目录下。启动httpd服务。

# systemctl enable httpd.service

# systemctl status httpd.service

 

1.4 配置ntp

# master

# yum -y install ntp

# vi /etc/ntp.conf

注释或者删除以下四行

server 0.centos.pool.ntp.org iburst

server 1.centos.pool.ntp.org iburst

server 2.centos.pool.ntp.org iburst

server 3.centos.pool.ntp.org iburst

添加以下两行

server 127.127.1.0

fudge 127.127.1.0 stratum 10

#systemctl enable ntpd

#systemctl start  ntpd

# slave1

# yum -y install ntpdate

# ntpdate master

# systemctl enable ntpdate

1.5 配置SSH

# master & slave1

检查2个节点是否可以通过无密钥相互访问，如果未配置，则进行SSH无密码公钥认证配置。如下：

# yum install openssh-clients

# ssh-keygen

# ssh-copy-id master.hadoop

# ssh-copy-id slave1.hadoop

ssh登录远程主机查看是否成功

# ssh master.hadoop

# exit

# ssh slave1.hadoop

# exit

1.6 禁用Transparent Huge Pages

操作系统后台有一个叫做khugepaged的进程，它会一直扫描所有进程占用的内存，在可能的情况下会把4kpage交换为Huge Pages，在这个过程中，对于操作的内存的各种分配活动都需要各种内存锁，直接影响程序的内存访问性能，并且，这个过程对于应用是透明的，在应用层面不可控制,对于专门为4k page优化的程序来说，可能会造成随机的性能下降现象。

# master & slave1

# cat /sys/kernel/mm/transparent_hugepage/enabled

[always] madvise never

# echo never > /sys/kernel/mm/transparent_hugepage/enabled

# echo never > /sys/kernel/mm/transparent_hugepage/defrag

# cat /sys/kernel/mm/transparent_hugepage/enabled

always madvise [never]

重启后失效，需要再次执行

1.7 安装配置JDK

# master

# mkdir /usr/jdk64/  

# mount -o loop XianDian-BigData-v2.2.iso /mnt/

# cd /mnt/

# tar -zxvf jdk-8u77-linux-x64.tar.gz -C /usr/jdk64/

# vi /etc/profile

export JAVA_HOME=/usr/jdk64/jdk1.8.0_77

export PATH=$JAVA_HOME/bin:$PATH

# source /etc/profile

#java –version

java version "1.8.0_77"

Java(TM) SE Runtime Environment (build 1.8.0_77-b03)

Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

# slave1

#mkdir /usr/jdk64/

#scp 172.24.14.106://mnt/jdk-8u77-linux-x64.tar.gz .

# tar -zxvf jdk-8u77-linux-x64.tar.gz -C /usr/jdk64/

# vi /etc/profile

export JAVA_HOME=/usr/jdk64/jdk1.8.0_77

export PATH=$JAVA_HOME/bin:$PATH

# source /etc/profile

#java –version

java version "1.8.0_77"

Java(TM) SE Runtime Environment (build 1.8.0_77-b03)

Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

2 配置ambari-server

# master

# yum -y install  ambari-server

2.1 安装MariaDB数据库

# master

# yum install  mariadb mariadb-server mysql-connector-java

启动服务

# systemctl enable mariadb

# systemctl start mariadb

配置MySQL

#mysql_secure_installation

按enter确认后设置数据库root密码，我们这里设置为“bigdata”

Remove anonymous users? [Y/n] y

Disallow root login remotely? [Y/n] n

Remove test database and access to it? [Y/n] y

Reload privilege tables now? [Y/n] y

创建ambari数据库

# mysql -uroot -pbigdata

MariaDB [(none)]> create database ambari;

MariaDB [(none)]> grant all privileges on ambari.* to ‘ambari‘@‘localhost‘ identified by ‘bigdata‘;

MariaDB [(none)]> grant all privileges on ambari.* to ‘ambari‘@‘%‘ identified by ‘bigdata‘;

MariaDB [(none)]> use ambari;

MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

MariaDB [ambari]> quit

 

2.2 安装配置ambari-server

# master

vi /etc/profile

export buildNumber=2.6.0.0

# ambari-server setup

WARNING: SELinux is set to ‘permissive‘ mode and temporarily disabled.

OK to continue [y/n] (y)?

Customize user account for ambari-server daemon [y/n] (n)? n

Checking JDK...

[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8

[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7

[3] Custom JDK

==============================================================================

Enter choice (1): 3

Path to JAVA_HOME: /usr/jdk64/jdk1.8.0_77

Validating JDK on Ambari Server...done.

Completing setup...

Configuring database...

Enter advanced database configuration [y/n] (n)? y

Configuring database...

====================================================================

Choose one of the following options:

[1] - PostgreSQL (Embedded)

[2] - Oracle

[3] - MySQL

[4] - PostgreSQL

[5] - Microsoft SQL Server (Tech Preview)

[6] - SQL Anywhere

====================================================================

Enter choice (1): 3

Hostname (localhost):

Port (3306):

Database name (ambari):

Username (ambari):

Enter Database Password (bigdata):

Proceed with configuring remote database connection properties [y/n] (y)?

Ambari Server ‘setup‘ completed successfully.

 

# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

 

启动ambari-server服务

# ambari-server start

登陆界面http://192.168.200.131:8080/

登录用户名密码为admin：admin

 

3 配置ambari-agent

# master & slave1

# yum -y install ambari-agent

# vi /etc/ambari-agent/conf/ambari-agent.ini

[server]

hostname= master.hadoop

# ambari-agent restart

# tail -f /var/log/ambari-agent/ambari-agent.log

INFO 2017-01-12 09:44:20,919 Controller.py:265 - Heartbeat response received (id = 1340)

INFO 2017-01-12 09:44:30,820 Heartbeat.py:78 - Building Heartbeat: {responseId = 1340, timestamp = 1484214270820, commandsInProgress = False, componentsMapped = True}