是否可以在 Hive 中的常量列上进行分组？

Posted 2023-04-18

【中文标题】是否可以在 Hive 中的常量列上进行分组？

【英文标题】：Is it possible to group by on a constant column in Hive?

【发布时间】：2017-04-17 07:08:02

【问题描述】：

我在标准非聚合表列 key1 和 key2 上运行 Hive 查询，如下所示。但我正在根据 where 条件向每个正在联合编辑的表添加一个常量类型列。

CREATE TABLE IF NOT EXISTS T_FINAL AS SELECT DISTINCT union_tbles.key1 AS key1, union_tbles.key2 AS key2, union_tbles.cnt AS cnt, union_tbles.type AS type FROM (
SELECT key1 AS key1, key2 AS key2, COUNT(val) AS cnt, 'x1' AS type FROM T_SUB1 WHERE key1 IN ('X1') GROUP BY key1, key2
UNION ALL
SELECT key1 AS key1, key2 AS key2, COUNT(val) AS cnt, 'x2' AS type FROM T_SUB1 WHERE key1 IN ('X2') GROUP BY key1, key2
) union_tbles

是否可以将常量列类型添加为分组列，如下所示？ 当我尝试在 group-by 中添加常量列类型时，我在 Hive 中收到 Invalid column alias 错误。任何建议如何在 Hive 中执行此操作？

CREATE TABLE IF NOT EXISTS T_FINAL AS SELECT DISTINCT union_tbles.key1 AS key1, union_tbles.key2 AS key2, union_tbles.cnt AS cnt, union_tbles.type AS type FROM (
SELECT key1 AS key1, key2 AS key2, COUNT(val) AS cnt, 'x1' AS type FROM T_SUB1 WHERE key1 IN ('X1') GROUP BY key1, key2, type
UNION ALL
SELECT key1 AS key1, key2 AS key2, COUNT(val) AS cnt, 'x2' AS type FROM T_SUB1 WHERE key1 IN ('X2') GROUP BY key1, key2, type
) union_tbles

【问题讨论】：

"SELECT DISTINCT"?

为什么是“UNION ALL”呢？

我正在对一系列 UNION 运行 JOIN。我以两种方式运行它 - SELECT DISTINCT ( ... UNION ALL ... UNION ALL ... etc ) 和 SELECT FROM ( ... UNION DISTINCT ... UNION DISTINCT )。第一种方式的作业数为 15，而第二种方式为 25。仅添加 UNION DISTINCT 即可显着增加作业数。每个作业使用 9068 个映射器和 1009 个减速器。所以它所花费的时间非常长。所以我想减少工作的数量。

但你首先不需要distinct（group by 后不能有重复项）也不需要union all。它可以使用单个select 编写。您正在尝试从其基础调整写得不好的内容。

附言。 select distinct from?是不是缺少一些列？

【参考方案1】：

Hive 无法识别 GROUP BY 子句中的别名。 无论如何，绝对不需要按常量分组。 常量不必在 GROUP BY 子句中才能被选中。

with t as (select 1 as x)
select    x
         ,count(*)
         ,1+1
         ,'Hello'
         ,current_date()
from      t
group by  x

---

+---+-----+-----+-------+------------+
| x | _c1 | _c2 |  _c3  |    _c4     |
+---+-----+-----+-------+------------+
| 1 |   1 |   2 | Hello | 2017-04-17 |
+---+-----+-----+-------+------------+