无法创建 InputInitializerManager 错误 - HIVE 上的 TEZ

Posted

技术标签:

【中文标题】无法创建 InputInitializerManager 错误 - HIVE 上的 TEZ【英文标题】:Failed to create InputInitializerManager error - TEZ on HIVE 【发布时间】:2016-07-28 16:33:05 【问题描述】:

我已经安装了 Apache Tez 0.8.1、Hadoop 版本 2.7.0 和 Hive 版本 2.01。我能够成功运行 Map Reduce Jobs。但是当我配置 hive 并尝试运行一个简单的计数查询时,它返回了下面的错误。从它试图查找 jar 的错误中,我已将 jar 放在类路径中,但仍然没有解决错误。

请帮助我解决这个问题。在此先感谢!

    hive> select count(*) from sample1;
    Query ID = root_20160728215555_a58e91a6-8913-4a57-8715-bc1739a2cb02
    Total jobs = 1
    Launching Job 1 out of 1


    ----------------------------------------------------------------------------------------------
        VERTICES      MODE        STATUS  TOTAL  COMPLETED  RUNNING  PENDING  FAILED  KILLED  
    ----------------------------------------------------------------------------------------------
    Map 1            container        FAILED     -1          0        0       -1       0       0  
    Reducer 2        container        KILLED      1          0        0        1       0       0  
    ----------------------------------------------------------------------------------------------
    VERTICES: 00/02  [>>--------------------------] 0%    ELAPSED TIME: 0.17 s     
    ----------------------------------------------------------------------------------------------
    Status: Failed
    Vertex failed, vertexName=Map 1, vertexId=vertex_1469720608711_0011_1_00, diagnostics=[Vertex vertex_1469720608711_0011_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
        at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:70)
        at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:89)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:138)
        at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:115)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4676)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.access$4300(VertexImpl.java:204)
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:3445)
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:3394)
        at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:3375)
        at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
        at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
        at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
        at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
        at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1975)
        at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:203)
        at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2090)
        at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:2076)
        at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
        at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
        at java.lang.Thread.run(Thread.java:745)
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:68)
        ... 20 more
    Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/split/SplitLocationProvider
        at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:96)
        ... 25 more
    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.split.SplitLocationProvider
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 26 more
    ]
    Vertex killed, vertexName=Reducer 2, vertexId=vertex_1469720608711_0011_1_01, diagnostics=[Vertex received Kill in NEW state., Vertex vertex_1469720608711_0011_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]
    DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1

更新:

在遇到上述问题后,我已将 hadoop-core-1.2.1.jar 复制到 hive lib 文件夹中。之后我在启动 hive 时遇到了另一个问题。从跟踪中我可以发现有一些通过了非法参数。

    Exception in thread "main" java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1550)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
        at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3080)
        at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3108)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:543)
        at org.apache.hadoop.hive.ql.session.SessionState.beginStart(SessionState.java:516)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:712)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
        at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1548)
        ... 14 more
    Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 1.2.1
        at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:165)
        at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:132)
        at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:93)
        at org.apache.hadoop.hive.metastore.ObjectStore.getDataSourceProps(ObjectStore.java:376)
        at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:268)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
        at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:517)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:482)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:544)
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:370)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
        at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
        at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5762)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:219)
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:67)

【问题讨论】:

任何 Tez 与 Hive 专家,请帮助我解决这个问题。我无法继续。 【参考方案1】:

问题在于 Apache Tez 版本。Apache Tez 0.8.1 与 Hadoop 2.7.0 和 hive 2.0.1 不兼容。

已下载 0.8.4 src 并构建它已解决问题。

谢谢!!。

【讨论】:

以上是关于无法创建 InputInitializerManager 错误 - HIVE 上的 TEZ的主要内容,如果未能解决你的问题,请参考以下文章

UE4无法创建C++项目

Android 错误:“无法创建 epoll 实例”或“无法创建唤醒管道”

虚拟桌面无法创建的问题

无法创建临时文件夹错误5拒绝访问

警告1909,无法创建快捷方式。怎么办?

创建EXCEL对象失败,请确认已正确安装Excel程序?