hadoop集群进行计算得不到结果

Posted 2023-03-02 肥大毛

使用hadoop集群hadoop102 hadoop103 hadoop104在统计hadoop-3.1.3/wcinput/word.txt中出现名字的次数时，
使用命令：

 hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /wcinput /wcoutput

在hdfs网页上未显示wcoutput计算结果

报错信息：
Exit code: 1
Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster

解决方案：

1、关闭yarn：在hadoop-3.1.3文件夹下执行 sbin/stop-yarn.sh
2、关闭hdfs：在hadoop-3.1.3文件夹下执行 sbin/stop-dfs.sh
3、修改mapred-site.xml：在hadoop-3.1.3文件夹下执行 vim etc/hadoop/mapred-site.xml
4、添加以下内容：
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/opt/module/hadoop-3.1.3</value>
</property>
再次执行命令：
 hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /wcinput /wcoutput