解决 mapreduce.Cluster 无法使用 org.apache.hadoop.mapred.YarnClientProtocolProvider 由于实例化 YarnClient 错误

Posted 2023-04-15

【中文标题】解决 mapreduce.Cluster 无法使用 org.apache.hadoop.mapred.YarnClientProtocolProvider 由于实例化 YarnClient 错误

【英文标题】：Resolving mapreduce.Cluster Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error in instantiating YarnClient

【发布时间】：2016-02-08 01:37:06

【问题描述】：

我在一个测试单节点集群上设置了一个基本的 Hadoop 2.6.0 + Hive 0.14.0，主要遵循以下指南：

Hadoop： http://tecadmin.net/setup-hadoop-2-4-single-node-cluster-on-linux/

蜂巢： https://cwiki.apache.org/confluence/display/Hive/GettingStarted

但是，如果我运行 WordCount java 示例，我会收到此错误：

$ hadoop jar wc.jar WordCount /user/hadoop/HDFSClient.java /user/hadoop/wc.out
16/02/08 12:24:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/02/08 12:24:56 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
16/02/08 12:24:56 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient
Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
        at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
        at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
        at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at WordCount.main(WordCount.java:59)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

conf/hdfs-site.xml:

<configuration>

<property>
 <name>dfs.replication</name>
  <value>1</value>
  </property>

<property>
  <name>dfs.name.dir</name>
      <value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
      </property>

<property>
  <name>dfs.data.dir</name>
      <value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
      </property>

<property>
  <name>dfs.namenode.http-address</name>
      <value>0.0.0.0:8000</value>
      </property>

</configuration>

conf/yarn-site.xml:

<configuration>

 <property>
   <name>yarn.nodemanager.aux-services</name>
       <value>mapreduce.shuffle</value>
        </property>

 <property>
   <name>yarn.resourcemanager.webapp.address</name>
     <value>0.0.0.0:8004</value>
        </property>
</configuration>

conf/core-site.xml：

<configuration>

<property>
  <name>mapred.job.tracker.http.address</name>
      <value>50031</value>
   </property>
<property>
  <name>fs.default.name</name>
      <value>hdfs://localhost:9000</value>
   </property>
</configuration>

我对设置 Hadoop 和 Hive 有点陌生，如果有任何关于可能需要查看的内容的建议或指示，我将不胜感激。

【问题讨论】：

检查你的 yarn-site.xml 属性...应该是mapreduce_shuffle

【参考方案1】：

用以下内容替换您的 core-site.xml、mapred-site.xml、hdfs-site.xml 和 yarn-site.xml 文件属性

核心站点.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>

hdfs-site.xml

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/namenode</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:///home/hadoop/hadoopdata/hdfs/datanode</value>
</property>
</configuration>

mapre-site.xml

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

yarn-site.xml

<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

【讨论】：

感谢您的回答。我试过了，但没有用。我已经通过 start/stop-yarn/dfs.sh 重新启动。运行 WordCount 示例时，我仍然遇到同样的错误。

Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses as your log file..mapred-site.xml file parameter is wrong..检查上述参数并再次使用更新的参数格式化您的namenode并启动您的守护进程..跨度>

我已经完成了重新格式化。这似乎有所帮助。但我仍然陷入同样的​​错误。我在 hadoop/logs/*.log 中执行了 WARN|ERROR 的 grep 操作，但看不到任何指示红旗的东西，所以我还在摸不着头脑。

如果我在 yarn-site 中有 yarn.nodemanager.aux-services=mapreduce_shuffle 会发出此警告：961 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: The Auxilurary Service named 'mapreduce_shuffle' in the configuration is for class class org.apache.hadoop.mapred.ShuffleHandler which has a name of 'httpshuffle'. Because these are not the same tools trying to send ServiceData and read Service Meta Data may have issues unless the refer to the name in the config.

在聊天中解决它对我没有帮助，因为我有同样的错误。

【参考方案2】：

如果您在类路径中使用“hadoop-mapreduce-client-jobclient.jar”运行应用程序，这似乎可以工作。