mapreduce执行时报java.net.ConnectException

Posted 2022-09-04 zhengzuozhanglina

 

mapreduce执行时报java.net.ConnectException

解决方案

在执行mapreduce程序时出现java.net.ConnectException: 拒绝连接异常,在错误信息中看到" localhost:35334"字眼,怀疑是没有配置主机名的原因, (1)尝试对每台虚拟机设置主机名,同时更新ssh密钥信息。
(2)更改hdfs-site.xml和yarn-site.xml相关内容,将原来的ip号改为主机名形式,并在./etc/hadoop/slaves文件下追加slave节点的主机名,将这三个文件分别传送到slave节点上。
(3)删除master下namenode和tmp路径下的所有信息,删除slave节点下nodename和tmp路径下的所有信息.对master节点重新进行format

./bin/hadoop namenode -format

 

配置 core-site.xml

<configuration>
<property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
 <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
        <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>
</configuration>

 

yarn-site.xml

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
        <description>host Single hostname that can be set in place of setting all yarn.resourcemanager*address resources. Results in default ports for ResourceManager components.</description>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
        <description></description>
    </property>
</configuration>

 

slaves

node1
node2

错误详细信息

[root@bogon bin]# ./hadoop jar /home/wordcount.jar /home/test.txt /home/test/rs.txt
19/08/27 23:33:17 INFO client.RMProxy: Connecting to ResourceManager at /192.168.120.128:8032
19/08/27 23:33:33 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/08/27 23:33:35 INFO input.FileInputFormat: Total input files to process : 1
19/08/27 23:33:35 INFO mapreduce.JobSubmitter: number of splits:1
19/08/27 23:33:35 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/08/27 23:33:36 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1566918823206_0003
19/08/27 23:33:36 INFO impl.YarnClientImpl: Submitted application application_1566918823206_0003
19/08/27 23:33:37 INFO mapreduce.Job: The url to track the job: http://bogon:8088/proxy/application_1566918823206_0003/
19/08/27 23:33:37 INFO mapreduce.Job: Running job: job_1566918823206_0003
19/08/27 23:43:02 INFO mapreduce.Job: Job job_1566918823206_0003 running in uber mode : false
19/08/27 23:43:02 INFO mapreduce.Job:  map 0% reduce 0%
19/08/27 23:43:03 INFO mapreduce.Job: Job job_1566918823206_0003 failed with state FAILED due to: Application application_1566918823206_0003
 failed 2 times due to Error launching appattempt_1566918823206_0003_000002. Got exception: java.net.ConnectException:
 Call From bogon/192.168.120.128 to localhost:35334 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.GeneratedConstructorAccessor44.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:754)
    at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1511)
    at org.apache.hadoop.ipc.Client.call(Client.java:1453)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
    at com.sun.proxy.$Proxy82.startContainers(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:128)
    at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
    at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
    at com.sun.proxy.$Proxy83.startContainers(Unknown Source)
    at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:122)
    at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:311)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: 拒绝连接
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:690)
    at org.apache.hadoop.ipc.Client$Connection.setupiostreams(Client.java:794)
    at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:412)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1568)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    ... 19 more
. Failing the application.
19/08/27 23:43:03 INFO mapreduce.Job: Counters: 0

 

参照文档

解决Host key verification failed