Hadoop2.7.3分布式搭建

Posted 2020-08-20

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Hadoop2.7.3分布式搭建相关的知识，希望对你有一定的参考价值。

一.从hadoop 下载2.7.3 安装包

版本：hadoop-2.7.3.tar.gz

下载地址：www.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz

百度云下载地址：

http://pan.baidu.com/s/1pLOyu9d 密码：s0j5

二. 解压下载好的hadoop-2.7.3.tar.gz到/usr/local目录下

$ sudo tar -xzvf hadoop-2.7.3.tar.gz

可以得到hadoop-2.7.3目录

三.hadoop配置

3.1 hadoop-env.sh

# cd /hadoop-2.7.3/etc/hadoop/
# sudo vim hadoop-env.sh
修改export JAVA_HOME=/usr/local/jdk1.8
yarn-env.sh (同上)
mapred-env.sh (同上)

3.2 slaves

删除 localhost

添加

hadoop1

hadoop2

3.2 core-site.xml

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://hadoop0:9000</value>
  </property>
  <property>
    <name>hadoop.tmp.dir</name>
    <value>file:/usr/local/hadoop-2.7.3/tmp</value>
  </property>
</configuration>

3.3 hdfs-site.xml

<configuration>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>/usr/local/hadoop-2.7.3/hdf/data</value>
    <final>true</final>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>/usr/local/hadoop-2.7.3/hdf/name</value>
    <final>true</final>
  </property>
</configuration>

3.4 mapred-site.xml

<configuration>
       <property>
              <name>mapreduce.framework.name</name>
              <value>yarn</value>
       </property>
       <property>
              <name>mapreduce.jobhistory.address</name>
              <value>hadoop0:10020</value>
       </property>
       <property>
              <name>mapreduce.jobhistory.webapp.address</name>
              <value>hadoop0:19888</value>
       </property>
</configuration>

3.5 yarn-site.xml

<configuration>      
       <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
              <value>org.apache.hadoop.mapred.ShuffleHandler</value>
       </property>
       <property>
              <name>yarn.resourcemanager.address</name>
              <value>hadoop0:8032</value>
       </property>
       <property>
              <name>yarn.resourcemanager.scheduler.address</name>
              <value>hadoop0:8030</value>
       </property>
       <property>
              <name>yarn.resourcemanager.resource-tracker.address</name>
              <value>hadoop0:8031</value>
       </property>
       <property>
              <name>yarn.resourcemanager.admin.address</name>
              <value>hadoop0:8033</value>
       </property>
       <property>
              <name>yarn.resourcemanager.webapp.address</name>
              <value>hadoop0:8088</value>
       </property>
</configuration>

四. 各个主机之间复制hadoop

# sudo scp -r /usr/local/hadoop-2.7.3 hadoop1:/usr/local/

# sudo scp -r /usr/local/hadoop-2.7.3 hadoop2:/usr/local/

五. 各个主机之间hadoop环境变量

5.1 # sudo vim /etc/profile

编辑内容：

export HADOOP_HOME=/usr/local/hadoop-2.7.3
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
export HADOOP_LOG_DIR=/usr/local/hadoop-2.7.3/logs
export YARN_LOG_DIR=$HADOOP_LOG_DIR