BigQuery:如何避免“查询执行期间超出资源”。错误
Posted
技术标签:
【中文标题】BigQuery:如何避免“查询执行期间超出资源”。错误【英文标题】:BigQuery: How to Avoid "Resources exceeded during query execution." error 【发布时间】:2015-09-22 19:34:34 【问题描述】:我想知道如何避免“执行期间超出资源”错误。关于此的大多数其他问题都涉及 JOIN EACH 或 GROUP EACH BY,但我已经没有使用这些了。如果我在日期或 ABS(HASH(userId)) 中包含 WHERE 子句,则查询有效,但我希望整个数据集可用,然后我将在 Tableau 中进一步过滤它。
如果我删除 t4 查询有效,但我想要最后一列,并且我希望从 event_parameters 字段中创建更多列以供以后查询。
作业 ID 是 rhi-localytics-db:job_6MaesvuMK6mP6irmAnrcM9R3cx8,以防万一,谢谢。
SELECT
t1.userId as userId,
t1.event_time AS event_time,
t1.Diamond_Balance as Diamond_Balance,
t2.Diamond_Change as Diamond_Change,
t3.Gold_Balance as Gold_Balance,
t4.Gold_Change as Gold_Change
FROM (
SELECT
userId,
event_time,
INTEGER(event_parameters.Value) AS Diamond_Balance,
FROM
FLATTEN([game_data], event_parameters)
WHERE
event_name LIKE 'Currency'
AND event_parameters.Name = 'Diamond_Balance'
-- and date(event_time) > '2015-09-11'
-- AND ABS(HASH(userId) % 5) = 0
GROUP BY
userId,
event_time,
Diamond_Balance ) AS t1
INNER JOIN (
SELECT
userId,
event_time,
INTEGER(event_parameters.Value) AS Diamond_Change,
FROM
FLATTEN([game_data], event_parameters)
WHERE
event_name LIKE 'Currency'
AND event_parameters.Name = 'Diamond_Change'
AND INTEGER(event_parameters.Value ) < 14000
AND INTEGER(event_parameters.Value ) > -14000
-- and date(event_time) > '2015-09-11'
-- AND ABS(HASH(userId) % 5) = 0
GROUP BY
userId,
event_time,
Diamond_Change ) AS t2
ON
t1.userId = t2.userId
AND t1.event_time = t2.event_time
INNER JOIN (
SELECT
userId,
event_time,
event_parameters.Value AS Gold_Balance,
FROM
FLATTEN([game_data], event_parameters)
WHERE
event_name LIKE 'Currency'
AND event_parameters.Name = 'Gold_Balance'
-- and date(event_time) > '2015-09-11'
-- AND ABS(HASH(userId) % 5) = 0
GROUP BY
userId,
event_time,
Gold_Balance ) AS t3
ON
t1.userId = t3.userId
AND t1.event_time = t3.event_time
INNER JOIN (
SELECT
userId,
event_time,
INTEGER(event_parameters.Value) AS Gold_Change,
FROM
FLATTEN([game_data], event_parameters)
WHERE
event_name LIKE 'Currency'
AND event_parameters.Name = 'Gold_Change'
-- and date(event_time) > '2015-09-11'
-- AND ABS(HASH(userId) % 5) = 0
GROUP BY
userId,
event_time,
Gold_Change ) AS t4
ON
t1.userId = t4.userId
AND t1.event_time = t4.event_time
【问题讨论】:
【参考方案1】:可在此处找到有关超出资源的一般建议: https://***.com/a/16579558/1375400
请注意,添加EACH
通常是资源超出错误的解决方案,而不是其原因。 (虽然在某些情况下它可以反过来工作!)
此外,EACH
在 GROUP BY
上不再有意义,并且很快将在 JOIN
上无关紧要。
【讨论】:
谢谢,我已经阅读了这个问题,但我一定错过了 EACH 是解决问题的方法。添加 JOIN EACH 使查询工作。【参考方案2】:我认为您应该能够在一次简单的“扫描”中完成所有逻辑。 根本没有加入! 像下面的东西。只是想法-但有可能按原样工作:)
SELECT
userId,
event_time,
MAX(CASE WHEN event_parameters.Name = 'Diamond_Balance'
THEN INTEGER(event_parameters.Value) END) AS Diamond_Balance,
MAX(CASE WHEN event_parameters.Name = 'Diamond_Change' AND INTEGER(event_parameters.Value ) BETWEEN -14000 AND 14000
THEN INTEGER(event_parameters.Value)) END AS Diamond_Change,
MAX(CASE WHEN event_parameters.Name = 'Gold_Balance'
THEN INTEGER(event_parameters.Value) END) AS Gold_Balance,
MAX(CASE WHEN event_parameters.Name = 'Gold_Change'
THEN INTEGER(event_parameters.Value) END) AS Gold_Change
FROM
FLATTEN([game_data], event_parameters)
WHERE
event_name LIKE 'Currency'
GROUP BY
userId,
event_time
【讨论】:
谢谢米哈伊尔,这也有效,是一个更简单的解决方案。我不确定它对于字符串而不是整数的 event_parameters 将如何工作。 只需删除 INTEGER()以上是关于BigQuery:如何避免“查询执行期间超出资源”。错误的主要内容,如果未能解决你的问题,请参考以下文章