Hadoop报错The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is excee
Posted DBArtist
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Hadoop报错The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is excee相关的知识,希望对你有一定的参考价值。
问题描述:调度系统执行hive任务失败,一直执行失败,报错如下:
java.io.IOException: java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
Caused by: java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
CONSOLE# Ended Job = job_1638255473937_0568 with exception \'java.io.IOException(java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused)
CONSOLE# FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. java.net.ConnectException: Call From #HostName/#IP to #HostName:10020 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
这个信息还看不出具体的问题所在,查看服务器上所有的日志,也没有看出问题,最后查看yarn的日志,看出问题所在。
根据调度系统,获取到ApplicationId:application_1638255473937_0568 , 然后从hdfs上查看对应的日志信息。
查看yarn日志信息:
[hdfs@centos hadoop27]$ yarn logs -applicationId application_1638255473937_0568
关键报错信息:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is exceeded: limit=1048576 items=1048576
以上是关于Hadoop报错The directory item limit of /tmp/hadoop-yarn/staging/history/done_intermediate/hdfs is excee的主要内容,如果未能解决你的问题,请参考以下文章
Hadoop HDFS: the directory item limit is exceed: limit=1048576问题的解决
Cannot lock storage /tmp/hadoop-root/dfs/name. The directory is already locked.
hadoop使用HDFS报错:ls: `.': No such file or directory
MySQL5.7报错–initialize specified but the data directory has files
MySQL5.7报错–initialize specified but the data directory has files
报错Hadoop bin directory does not exist: E:softwawrehadoop3.xhadoop-3.1.0inin -see