Amabari搭建Hadoop集群
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Amabari搭建Hadoop集群相关的知识,希望对你有一定的参考价值。
一、系统环境
1.系统版本
[[email protected] ~]# cat /etc/centos-release CentOS Linux release 7.4.1708 (Core)
2.主机规划
主机名 | IP地址 | 角色 |
manager | 192.168.10.131 | ambari-server |
vm1 | 192.168.10.128 | agent |
vm2 | 192.168.10.129 | agent |
vm3 | 192.168.10.130 | agent |
3.软件版本
ambari-2.4.3.0-centos7.tar.gz HDP-2.5.3.0-centos7-rpm.tar.gz HDP-UTILS-1.1.0.21-centos7.tar.gz
二、环境准备(所有节点上都要配置)
1.修改主机名
2.防火墙和selinux
[[email protected] ~]# systemctl status firewalld ● firewalld.service - firewalld - dynamic firewall daemon Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:firewalld(1) [[email protected] ~]# getenforce Disabled [[email protected] ~]#
3.配置hosts文件
[[email protected] ~]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.10.128 vm1 192.168.10.129 vm2 192.168.10.130 vm3 192.168.10.131 manager
4.配置SSH免密码登陆
5.关闭linux的THP服务
#在GRUB_CMDLINE_LINUX这一行后面添加transparent_hugepage=never,如下 [[email protected] ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet transparent_hugepage=never" GRUB_DISABLE_RECOVERY="true" #执行以下命令 [[email protected] ~]# grub2-mkconfig -o /boot/grub2/grub.cfg [[email protected] ~]# systemctl disable tuned #重启后显示如下 [[email protected] ~]# cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] [[email protected] ~]# grep AnonHugePages /proc/meminfo AnonHugePages: 0 kB [[email protected] ~]#
6.配置UMASK
[[email protected] ~]# umask 0022 [[email protected] ~]# umask 0022
7.配置时间同步NTP
[[email protected] ~]# yum -y install ntp [[email protected] ~]# systemctl start ntpd [[email protected] ~]# systemctl enable ntpd
8.安装JDK并配置JAVA_HOME环境变量
三、安装Ambari集群(只在manager上配置)
1.安装HTTP服务
[[email protected] ~]# yum install -y httpd [[email protected] ~]# systemctl start httpd [[email protected] ~]# systemctl enable httpd
2.安装mysql
3.配置MySQL
#mysql有密码复杂度要求,如果不需要密码策略在/etc/my.cnf文件中添加如下配置并重启mysql即可禁用 validate_password = off systemctl restart mysqld #查看mysql的初始默认密码 [[email protected] ~]# cat /var/log/mysqld.log | grep password 2017-12-13T01:49:55.629403Z 1 [Note] A temporary password is generated for [email protected]: 3aYp&Zsqq7Ov #使用初始密码进入mysql修改密码 ALTER USER 'root'@'localhost' IDENTIFIED BY '123456'; #在mysql中创建ambari数据库及用户 CREATE DATABASE ambari CHARACTER SET utf8 ; CREATE USER 'ambari'@'%'IDENTIFIED BY '123456'; CREATE USER 'ambari'@'local'IDENTIFIED BY '123456'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; FLUSH PRIVILEGES; #创建hive和其它数据库也类似
4.下载Ambari和HDP
wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.4.3.0/ambari-2.4.3.0-centos7.tar.gz wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0/HDP-2.5.3.0-centos7-rpm.tar.gz wget -nv http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.21/repos/centos7/HDP-UTILS-1.1.0.21-centos7.tar.gz
5.解压至httpd目录下
[[email protected] ~]# mkdir /var/www/html/ambari [[email protected] ~]# mkdir /var/www/html/ambari/HDP-UTILS tar zxvf ambari-2.4.3.0-centos7.tar.gz -C /var/www/html/ambari tar zxvf HDP-2.5.3.0-centos7-rpm.tar.gz -C /var/www/htmlambari tar zxvf HDP-UTILS-1.1.0.21-centos7.tar.gz -C /var/www/html/ambari/HDP-UTILS
6.安装本地源制作相关工具
[[email protected] ~]# yum install -y yum-utils createrepo yum-plugin-priorities [[email protected] ~]# vim /etc/yum/pluginconf.d/priorities.conf [main] enabled = 1 gpgcheck=0 [[email protected] data]# cd /var/www/html/ambari [[email protected] ambari]# createrepo ./
7.配置本地源
[[email protected] ~]# cat /etc/yum.repos.d/ambari.repo [ambari-2.4.3.0] name=ambari-2.4.3.0 baseurl=http://manager/ambari/AMBARI/ gpgcheck=1 gpgkey=http://manager/ambari/AMBARI/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [[email protected] ~]# cat /etc/yum.repos.d/hdp.repo [HDP-2.5.3.0] name=HDP baseurl=http://manager/ambari/HDP/centos7/ gpgcheck=1 gpgkey=http://manager/ambari/HDP/centos7/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.21] name=HDP-UTILS baseurl=http://manager/ambari/HDP-UTILS/ gpgcheck=1 gpgkey=http://manager/ambari/HDP-UTILS/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1
[[email protected] ~]# yum clean all [[email protected] ~]# yum makecache #查看ambari和hdp的资源库 [[email protected] ~]# yum repolist repo id repo name status HDP-2.5.3.0 HDP 200 HDP-UTILS-1.1.0.21 HDP-UTILS 64 ambari-2.4.3.0 ambari-2.4.3.0
8.安装ambari和mysql驱动
[[email protected] ~]# yum install -y ambari [[email protected] ~]# yum install -y mysql-connector-java
9.配置ambari
[[email protected] ~]# ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' Customize user account for ambari-server daemon [y/n] (n)? y Enter user account for ambari-server daemon (root):root Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... Do you want to change Oracle JDK [y/n] (n)? y [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7 [3] Custom JDK ============================================================================== Enter choice (1): 3 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. Path to JAVA_HOME: /usr/local/jdk Validating JDK on Ambari Server...done. Completing setup... Configuring database... Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== Enter choice (3): 3 Hostname (localhost): Port (3306): Database name (ambari): Username (ambari): Enter Database Password (123456): Configuring ambari database... Copying JDBC drivers to server resources... Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... ............. Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully.
10.使用msyql用户ambari在ambari数据库中导入数据
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
11.启动ambari
[[email protected] ~]# ambari-server start #在浏览器中输入ambari地址 http://192.168.10.131:8080
以上是关于Amabari搭建Hadoop集群的主要内容,如果未能解决你的问题,请参考以下文章
CentOS5/6/7系统下搭建安装Amabari大数据集群时出现SSLError: Failed to connect. Please check openssl library versions.