当查询具有 GROUP BY 子句时,Hive 查询抛出“来自 org.apache.hadoop.hive.ql.exec.tez.TezTask 的代码 2”异常

Posted

技术标签:

【中文标题】当查询具有 GROUP BY 子句时,Hive 查询抛出“来自 org.apache.hadoop.hive.ql.exec.tez.TezTask 的代码 2”异常【英文标题】:Hive query throw "code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask" exception when query has GROUP BY cluase 【发布时间】:2021-04-06 09:54:09 【问题描述】:

我在 HDP 3.1.4 上有 Hive + LLAP

Hive 和 Tez 配置是:

yarn.nodemanager.resource.memory-mb = 40960
yarn.scheduler.minimum-allocation-mb = 1024
yarn.scheduler.maximum-allocation-mb = 40960
hive.tez.container.size = 4096
num_llap_nodes=4
hive.llap.daemon.num.executors=8
hive.llap.daemon.yarn.container.mb = 35840
llap_headroom_space=2048
llap_heap_size=32768
hive.llap.io.memory.size=1024
tez.am.resource.memory.mb=4096
hive.tez.java.opts=-server -Djava.net.preferIPv4Stack=true -XX:NewRatio=8 -XX:+UseNUMA -XX:+UseG1GC -XX:+ResizeTLAB -XX:+PrintGCDetails -verbose:gc -XX:+PrintGCTimeStamps **-Xmx3276m**
tez.runtime.io.sort.mb= 1638
tez.runtime.unordered.output.buffer.size-mb=409

以下查询运行正常:

select count(*) from balance;

但在以下查询中使用 group by 表达式时:

select count(*),jobdate from balance group by jobdate;

我 我尝试了很多配置,但抛出了这个很长的异常:

ERROR: Error while processing statement: **FAILED: Execution Error, 
    return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask.** 
    Vertex failed, vertexName=Map 1, 
    vertexId=vertex_1617520101397_0014_1_00, diagnostics=[Task 
    failed, taskId=task_1617520101397_0014_1_00_000013, 
    diagnostics=[TaskAttempt 0 failed, **info=[Error: Error while 
    running task ( failure ) : java.lang.NoClassDefFoundError: Could 
    not initialize class 
    org.apache.tez.runtime.library.api.TezRuntimeConfiguration**    at 

    **BLABLA**
        at java.lang.Thread.run(Thread.java:748) ]], Task failed, 
    taskId=task_1617520101397_0014_1_00_000006, 
    diagnostics=[TaskAttempt 0 failed, info=[Error: Error while 
    running task ( failure ) : java.lang.NoClassDefFoundError: Could 
    not initialize class 
    org.apache.tez.runtime.library.api.TezRuntimeConfiguration  at 
        at java.lang.Thread.run(Thread.java:748) ]], Task failed, 
    taskId=task_1617520101397_0014_1_00_000005, 
    diagnostics=[TaskAttempt 0 failed, info=[Error: Error while 
    running task ( failure ) : java.lang.NoClassDefFoundError: Could 
    not initialize class 
    org.apache.tez.runtime.library.api.TezRuntimeConfiguration  at 
    org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:111) 
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
        at java.lang.Thread.run(Thread.java:748) ]], **Vertex did not 
    succeed due to OWN_TASK_FAILURE, failedTasks:9 killedTasks:31761, 
    Vertex vertex_1617520101397_0014_1_00 [Map 1] killed/failed due 
    to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, 
    vertexId=vertex_1617520101397_0014_1_01, diagnostics=[Vertex 
    received Kill while in RUNNING state., Vertex did not succeed due 
    to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:18, Vertex 
    vertex_1617520101397_0014_1_01 [Reducer 2] killed/failed due 
    to:OTHER_VERTEX_FAILURE]DAG did not succeed due to 
    VERTEX_FAILURE. failedVertices:1 killedVertices:1 Error Code: 2**

【问题讨论】:

我根据以下参考尝试了许多配置:community.cloudera.com/t5/Community-Articles/…community.cloudera.com/t5/Community-Articles/… 【参考方案1】:

在 Ambari Hive 配置页面中有两个部分用于设置 hive.tez.container.size。其中一个出现在 SETTINGS 选项卡中,另一个与 LLAP 相关的位于 ADVANCED 选项卡的 Advanced hive-interactive-site 下.我尝试使用 hive.tez.container.size 值设置选项卡而不是 高级 hive-interactive-site 部分。最后,我设置了以下配置并解决了错误:

set hive.tez.container.size=10240;
set hive.tez.java.opts=-Xmx9216m;
set tez.runtime.io.sort.mb=3072;
set tez.runtime.unordered.output.buffer.size-mb=1024;

【讨论】:

你在哪里做设置部分?在 localhost:4200 shell 或 Ambari Hive 配置 UI 中。

以上是关于当查询具有 GROUP BY 子句时,Hive 查询抛出“来自 org.apache.hadoop.hive.ql.exec.tez.TezTask 的代码 2”异常的主要内容,如果未能解决你的问题,请参考以下文章

Hive SQL子句中 group by 1 是什么意思?

我们如何在具有GROUP BY子句的查询中选择非聚合列,而GROUP BY子句在功能上不依赖于GROUP BY子句中的列?

JPA Group by 具有多个字段

group by 子句中的 sparkSQL Map 列

如何使用 group by(基于一列)从表中选择多列,在 hive 查询中具有和计数

在 WHERE 或 GROUP BY 子句中使用列表别名