bigquery 分区表的分区修剪

Posted 2023-03-24

【中文标题】bigquery 分区表的分区修剪

【英文标题】：Partition pruning for bigquery partitioned table

【发布时间】：2018-03-12 18:14:55

【问题描述】：

我有一个使用分析函数查询天分区表的查询。我希望它只读取在 where 子句中过滤的分区中的数据，但它会读取表中的所有分区。

WITH query AS (
SELECT
  * EXCEPT(rank)
FROM (
  SELECT
    *,
    RANK() OVER (PARTITION BY day order by num_mean_temp_samples) AS rank
  FROM (
    SELECT
      FORMAT_DATE("%Y%m%d", _PARTITIONDATE) AS day,
      *
    FROM
      `mydataset.gsod_partitioned` ) q_nested
  ) q
WHERE
  rank < 1000
)
SELECT
  num_mean_temp_samples ,
  count(1) as samples
FROM query
 WHERE
   day in ( '20100101', '20100103')
GROUP BY  1 ORDER BY 1

我验证了没有解析函数的分区修剪工作：

WITH query AS (
SELECT
  FORMAT_DATE("%Y%m%d", _PARTITIONDATE) AS day,
  *
FROM
  `mydataset.gsod_partitioned`
)

或在添加 UNION ALL 嵌套选择之后：

WITH query AS (
SELECT
  * EXCEPT(rank)
FROM (
  SELECT
    *,
    RANK() OVER (PARTITION BY day order by num_mean_temp_samples) AS rank
  FROM (
    SELECT
      FORMAT_DATE("%Y%m%d", _PARTITIONDATE) AS day,
      *
    FROM
      `mydataset.gsod_partitioned` WHERE _PARTITIONDATE < "1970-01-01" ) q_nested1
  UNION ALL SELECT
    *,
    RANK() OVER (PARTITION BY day order by num_mean_temp_samples) AS rank
  FROM (
    SELECT
      FORMAT_DATE("%Y%m%d", _PARTITIONDATE) AS day,
      *
    FROM
      `mydataset.gsod_partitioned` WHERE _PARTITIONDATE >= "1970-01-01" ) q_nested2
  ) q
WHERE
  rank < 1000
)

表 mydataset.gsod_partitioned 是基于 gsod 的公共数据集，其中 day=20100101 分区创建如下：

bq query --destination_table 'private.gsod_partitioned$20100101' --time_partitioning_type=DAY --use_legacy_sql=false
'SELECT station_number, mean_temp, num_mean_temp_samples FROM `bigquery-public-data.samples.gsod` where year=2010 and month=01 and day=01'

您能否找到一种方法来为分析函数启用分区修剪而不向查询添加额外的联合？

【参考方案1】：

关于 _PARTITIONDATE - 它没有记录的功能，建议改用 _PARTITIONETIME，您可以寻找其他一些问题来查看其中一位 Google 员工的说法：Use of the _PARTITIONDATE vs. the _PARTITIONTIME pseudo-columns in BigQuery

关于partition pruning with analitycal functions，Google 在去年增加了对过滤器下推的支持，但仅适用于 _PARTITIONTIME，它应该包含在 PARTITON BY 子句涵盖的字段中

应该是这样的：

WITH query AS (
SELECT
  * EXCEPT(rank)
FROM (
  SELECT
    *,
    RANK() OVER (PARTITION BY _pt order by num_mean_temp_samples) AS rank
  FROM (
    SELECT
      FORMAT_TIMESTAMP("%Y%m%d", _PARTITIONTIME) AS day,
      _PARTITIONTIME as _pt,
      *
    FROM
      `mydataset.gsod_partitioned` ) q_nested
  ) q
WHERE
  rank < 1000
)
SELECT
  num_mean_temp_samples ,
  count(1) as samples
FROM query
 WHERE
   day in ( '20100101', '20100103')
GROUP BY  1 ORDER BY 1

【讨论】：

感谢您的回答。我验证了使用 FORMAT_TIMESTAMP("%Y%m%d", _PARTITIONTIME) 函数更改 _PARTITIONDATE -> _PARTITIONTIME 也不会修剪分区。在不格式化的情况下使用 _PARTITIONDATE 或 _PARTITIONTIME 可以启用分区修剪，但我需要格式化日期。

我明白你的意思。请查看添加的代码，它应该适合您

这是一个很好的解决方法，但仍然认为它应该适用于格式化的一天，不是吗？

是的，让它像这样工作会很棒，但我不会认为它是一个错误。对我来说，它有资格在 issuetracker.google.com/issues/new?component=187149 提出功能请求，也许 BigQuery 团队的某个人会在这里回复您的建议。

似乎谓词“day in ('20100101', '20100103')”没有被下推，所以分区没有被修剪。在 UNION ALL 版本中，您将谓词放在表的右侧，_PARTITIONDATE