搭建Hadoop集群Tips

Posted 2020-07-21 啦啦小明

7 YARN安装过程

 在安装HDFS的基础上安装YARN

1）修改mapred-site.xml文件，配置MapReduce

 

  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>master:10020</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>master:19888</value>
  </property>

2）修改yarn-env.sh

修改JAVA_HOME值（export JAVA_HOME=/usr/local/java/jdk1.7.0_79）

3）修改yarn-site.xml

配置ResourceManager

  <property>
    <name>yarn.resourcemanager.address</name>
    <value>master:8032</value>
  </property>
  <property>
    <name>yarn.resourcemanager.scheduler.address</name>
    <value>master:8030</value>
  </property>
  <property>
    <name>yarn.resourcemanager.resource-tracker.address</name>
    <value>master:8031</value>
  </property>
  <property>
    <name>yarn.resourcemanager.admin.address</name>
    <value>master:8033</value>
  </property>
  <property>
    <name>yarn.resourcemanager.webapp.address</name>
    <value>master:8088</value>
  </property>

配置NodeManager

  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>master</value>  
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
  </property>

4）启动YARN

root@master:/usr/local/hadoop# start-yarn.sh

master有如下进程：

slaves有如下进程：

5）运行wordcount验证

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar wordcount /input /output

运行jar包的命令可参考官网，如下图：

运行截图如下：

查看结果

至此，hadoop集群已全部配置完成。