使用 rank() 时未能将窗口调用分解为组

Posted 2023-04-18

【中文标题】使用 rank() 时未能将窗口调用分解为组

【英文标题】：Failing to breakup Windowing invocations into Groups when using rank()

【发布时间】：2020-08-27 13:45:05

【问题描述】：

我有一个属性数据集，我正在尝试按每个段的属性计数对其进行排名。然后，我想将具有大多数属性的段分配为管理这些属性的 xml_id 的段。我试过这个：

select
    yyyy_mm_dd,
    xml_id,
    ps_segment
from(
    select
        pspd.yyyy_mm_dd,
        xppd.xml_id,
        ps.ps_segment,
        count(pspd.property_id) as property_count.
        rank() over (partition by pspd.yyyy_mm_dd order by count(pspd.property_id) desc) rn 
    from(
        select
            yyyy_mm_dd,
            property_id
        from
            t1
    ) pspd
    left join(
        select
            yyyy_mm_dd,
            property_id,
            xml_id
        from
            t2
    ) xppd on xppd.property_id = pspd.property_id and xppd.yyyy_mm_dd = pspd.yyyy_mm_dd
    inner join
        t3 ps on ps.property_id = property_id
    group by
        1,2,3
) x
where
    rn = 1

上面抛出以下错误：

编译语句时出错：FAILED: SemanticException 无法将窗口调用分解为组。至少 1 个组必须仅依赖于输入列。还要检查循环依赖。潜在错误：org.apache.hadoop.hive.ql.parse.SemanticException: Line 5:50 Expression not in GROUP BY key 'property_count'

所以基本上xml_id 以拥有最高属性数的ps_segment 结束。我在查询中做错了什么？

【参考方案1】：

我自己解决了这个问题，下面的查询有效。看来我应该使用ROW_NUMBER() 而不是RANK()：

select
    yyyy_mm_dd,
    xml_id,
    ps_segment
from(
    select
        pspd.yyyy_mm_dd,
        xppd.xml_id,
        ps.ps_segment,
        count(pspd.property_id) as property_count,
        ROW_NUMBER() over (partition by pspd.yyyy_mm_dd, xml_id order by count(pspd.property_id) desc) rn
    from(
        select
            yyyy_mm_dd,
            property_id
        from
            t1
        where
            and yyyy_mm_dd = '2019-10-10'
    ) pspd
    left join(
        select
            yyyy_mm_dd,
            property_id,
            xml_id
        from
            t2
        where
            yyyy_mm_dd = '2019-10-10'
    ) xppd on xppd.property_id = pspd.property_id and xppd.yyyy_mm_dd = pspd.yyyy_mm_dd
    inner join
        t3 ps on ps.hotel_id = pspd.hotel_id
    group by
        1,2,3
) x
where
    rn = 1