Hadoop2.7.2安装与集群搭建

Posted 2021-01-18 观离

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Hadoop2.7.2安装与集群搭建相关的知识，希望对你有一定的参考价值。

1.环境准备

jdk需要1.7以上版本64位.
创建hadoop用户.
在hadoop用户目录下解压安装包hadoop-2.7.2.tar.gz

2.配置免密码登录

各节点分别执行

生成公钥和私钥:ssh-keygen -t rsa
四次enter.

将公钥添加进公钥库:cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
修改authorized_keys权限:chmod 600 ~/.ssh/authorized_keys
验证:ssh localhost

拷贝

将各节点中id_rsa.pub中的内容共同拷贝到authorized_keys,将authorized_keys覆盖各节点.

3.创建目录

mkdir -p ~/hadoop/dfs/name
mkdir -p ~/hadoop/dfs/data
mkdir -p ~/hadoop/tmp

4.修改配置文件

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
       <name>hadoop.tmp.dir</name>
       <value>/home/wangkai/hadoop/tmp</value>
</property>
<property>
       <name>fs.defaultFS</name>
       <value>hdfs://centos7-1:9000</value>
</property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/home/wangkai/hadoop/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/home/wangkai/hadoop/dfs/data</value>
    </property>
</configuration>

yarn-site.xml

<?xml version="1.0"?>
<configuration>
 <property> 
 <name>mapreduce.framework.name</name> 
 <value>yarn</value> 
 </property> 
 <property> 
 <name>yarn.nodemanager.aux-services</name> 
 <value>mapreduce_shuffle</value> 
 </property> 
<property> 
 <name>yarn.resourcemanager.hostname</name> 
 <value>192.168.163.101</value> 
 </property> 
</configuration>

注:不配置yarn.resourcemanager.hostname可能会导致从节点的nodemanager无法与主节点resourcemanager通信,默认为0.0.0.0,需要修改.

slaves

centos7-1
centos7-2
centos7-3

hadoop-env.sh

第一行添加

export JAVA_HOME=/usr/local/jdk1.7.0_79

yarn-env.sh

第一行添加:

export JAVA_HOME=/usr/local/jdk1.7.0_79

5.修改系统文件(各节点)

/etc/hosts

文件末尾添加

192.168.163.101 centos7-1
192.168.163.102 centos7-2
192.168.163.103 centos7-3

~/.bashrc

文件末尾添加

export JAVA_HOME=/usr/local/jdk1.7.0_79
export HADOOP_HOME=/home/wangkai/hadoop
export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

source ~/.bashrc