hdfs目录存储最大文件数异常MaxDirectoryItemsExceededException

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了hdfs目录存储最大文件数异常MaxDirectoryItemsExceededException相关的知识,希望对你有一定的参考价值。

问题描述

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException): The directory item limit of /XXX/XXX/FF is exceeded: limit=1048576 items=1048576

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:2060)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:2112)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addLastINode(FSDirectory.java:2081)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addINode(FSDirectory.java:1900)

        at org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:327)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2794)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2674)

        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2559)

        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:592)

        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.create(AuthorizationProviderProxyClientProtocol.java:110)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:395)

        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)

        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)

        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

 

        at org.apache.hadoop.ipc.Client.call(Client.java:1475)

        at org.apache.hadoop.ipc.Client.call(Client.java:1412)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)

        at com.sun.proxy.$Proxy10.create(Unknown Source)

        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)

        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)

        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)

        at com.sun.proxy.$Proxy11.create(Unknown Source)

        at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)

        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)

        at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)

        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)

        at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)

        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)

        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)

        at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)

        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)

        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)

        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)

        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)

        at com.cnpc.lhjd.Apps604$.com$cnpc$lhjd$Apps604$$readImgFromDb$1(Apps604.scala:104)

        at com.cnpc.lhjd.Apps604$$anonfun$main$1.apply(Apps604.scala:567)

        at com.cnpc.lhjd.Apps604$$anonfun$main$1.apply(Apps604.scala:562)

        at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)

        at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)

        at com.cnpc.lhjd.Apps604$.main(Apps604.scala:562)

        at com.cnpc.lhjd.Apps604.main(Apps604.scala)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at scala.reflect.internal.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:70)

        at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)

        at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:101)

        at scala.reflect.internal.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:70)

        at scala.reflect.internal.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)

        at scala.tools.nsc.CommonRunner$class.run(ObjectRunner.scala:22)

        at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:39)

        at scala.tools.nsc.CommonRunner$class.runAndCatch(ObjectRunner.scala:29)

        at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:39)

        at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:65)

        at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:87)

        at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:98)

        at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103)

        at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

 

意思是在当前hdfs目录下超过了最大可容纳文件个数,默认是1048576

 

解决办法

更改hdfs-site.xml添加如下

<property>
  <name>dfs.namenode.fs-limits.max-directory-items</name>
  <value>1048576</value>
  <description>Defines the maximum number of items that a directory may
      contain. Cannot set the property to a value less than 1 or more than
      6400000.</description>
</property>

把这个配置添加到hdfs-site.xml中,把值设置为大一些,问题搞定。

不过在此也存在一个问题,这个HDFS的限制有个范围,最多不能超过6400000,因此后续还要考虑到历史数据的删除。

 

 

以上是关于hdfs目录存储最大文件数异常MaxDirectoryItemsExceededException的主要内容,如果未能解决你的问题,请参考以下文章

根据 hdfs 目录中的文件名计算文件数

ulimit 设置 最大 打开文件数(祝大家 未来 精彩无限 -- unlimited)

JavaScript 中 FileReader#readEntries 可以读取的目录的最大文件数

从 Spark 估计要写入 HDFS 的部分文件数

centos7,进程最大打开文件数 too many open files错误

hadoop hdfs设置quota