当优化器参数更改（ON/OFF）时，HAWQ 为相同的 sql 返回不同的输出
Posted
技术标签:
【中文标题】当优化器参数更改（ON/OFF）时，HAWQ 为相同的 sql 返回不同的输出【英文标题】：HAWQ returns different output for same sql when optimizer parameter is changed (ON/OFF) 【发布时间】：2015-02-22 08:34:33 【问题描述】：在开发 GreenPlum HAWQ 1.1.3 时遇到了一个奇怪的行为。 想要执行一个简单的exists查询，但得到了错误的结果。
gpadmin=# select version();
version
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
PostgreSQL 8.2.15 (Greenplum Database 4.2.0 build 1) (HAWQ 1.1.3.0 build 4609) on x86_64-unknown-linux-gnu, compiled by GCCgcc (GCC) 4.4.2 compiled on Oct 27 2013 20:
53:09
(1 row)
gpadmin=# Set optimizer=off;
SET
gpadmin=# with temp as (
Select 1 one, 2 two
union all
Select 3,4
),
temp1 as (
Select 3 three
)
Select * from temp
where exists (Select * from temp1 where three=one);
one | two
-----+-----
(0 rows)
gpadmin=#
因此，当 HAWQ 的 优化器参数 设置为 OFF 时，查询返回 0 行，理想情况下应该返回存在于 temp1 和 temp 数据集中的记录，即 3,4.
所以我尝试了相同的查询，但将参数设置为 ON
gpadmin=# Set optimizer=on;
SET
gpadmin=#
with temp as (
Select 1 one, 2 two
union all
Select 3,4
),
temp1 as (
Select 3 three
)
Select * from temp
where exists (Select * from temp1 where three=one);
one | two
-----+-----
3 | 4
(1 row)
这里查询返回了预期的结果。
猜想 Greenplum 工程师应该想出一个解决方案来解决他们的优化器的这种错误行为。
当上述查询在 Greenplum DCA 4.2 上执行时，数据库崩溃并重新启动。我不明白如何向世界发布这种不成熟/未经测试的产品。这些只是一些简单的基本查询，在推向市场之前至少应该经过测试（alpha 测试）。最重要的是，当他们的工程师团队提出这个问题时，他们身边的支持人员要求为上述查询中的表提供 DDL。 我猜支持人员没有理解这个查询不包含任何表，它只包含一个使用 with 子句创建的临时数据集。
【问题讨论】：
【参考方案1】：-
“优化器”与更深层次的查询优化无关。它启用了名为
ORCA 的全新查询优化器，而在禁用状态下使用的Postgres 优化器称为planner
应将此问题提交给 Pivotal Support，然后由工程师解决
这是由union all 引起的极端情况，并且在 master 上选择，它适用于表格
代码如下：
create table test (one int, two int);
insert into test values (1, 2), (3, 4);
create table test2 (three int);
insert into test2 values (3);
Select * from test
where exists (Select * from test2 where three=one);
【讨论】：
【参考方案2】：我稍微改写了您的查询，将 temp 重命名为 t1，并将 temp1 重命名为 t2，以便更清楚地解释问题。
您的问题可以分为两个方面来回答：
1) 我想您使用的是旧版本的 hawq，其中查询在优化器打开时工作，而在优化器关闭时不工作。 在最新的apache hawq 中，它实际上不适用于优化器的开/关。原因是当优化器打开时它现在回退到规划器。
2) 查询失败是由于查询执行器的错误造成的。
具体来说，把CTE（公用表表达式）子句中的t2当做子查询扫描然后物化是对的。
但是，在计算 t2 时，它不会生成元组。因此，t1 与 t2 的连接不会生成元组。
我们可以在查询执行统计中看到这一点，同时使用优化器 = 关闭运行查询时使用解释分析。
-> Materialize (cost=0.00..0.01 rows=1 width=0)
Rows out: 0 rows with 0.167 ms to end of 3 scans, start offset by 0.265 ms.
-> Limit (cost=0.00..0.00 rows=1 width=0)
Rows out: 0 rows with 0.003 ms to end, start offset by 0.257 ms.
-> Subquery Scan t2 (cost=0.00..0.01 rows=1 width=0)
Rows out: 0 rows with 0.002 ms to end, start offset by 0.258 ms.
-> Result (cost=0.00..0.01 rows=1 width=0)
One-Time Filter: 3 = $0
Rows out: 0 rows with 0.001 ms to end, start offset by 0.258 ms.
我们有HAWQ-884 跟踪此问题，您可以参考它以获取更新和详细信息。
PS：这里是查询执行统计的详细信息：
1) 关闭优化器的 hawq 2.0（规划器）
show optimizer;
optimizer
-----------
off
(1 row)
WITH t1 AS ( SELECT 1 c1, 2 c2 UNION ALL SELECT 3 c1, 4 c2 ),
t2 AS ( SELECT 3 c3 )
SELECT * FROM t1
WHERE EXISTS (SELECT * FROM t2 WHERE c3=c1);
c1 | c2
----+----
(0 rows)
EXPLAIN ANALYZE
WITH t1 AS ( SELECT 1 c1, 2 c2 UNION ALL SELECT 3 c1, 4 c2 ),
t2 AS ( SELECT 3 c3 )
SELECT * FROM t1
WHERE EXISTS (SELECT * FROM t2 WHERE c3=c1);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.05..0.29 rows=72 width=8)
Rows out: Avg 0.0 rows x 0 workers. Max/Last(/) 0/0 rows with 0.237/0.237 ms to end.
-> Limit (cost=0.00..0.00 rows=1 width=0)
Rows out: Avg 0.0 rows x 0 workers. Max/Last(/) 0/0 rows with 0.003/0.003 ms to end.
-> Subquery Scan t2 (cost=0.00..0.01 rows=6 width=0)
Rows out: Avg 0.0 rows x 0 workers. Max/Last(/) 0/0 rows with 0.002/0.002 ms to end.
-> Result (cost=0.00..0.01 rows=1 width=0)
One-Time Filter: 3 = $0
Rows out: Avg 0.0 rows x 0 workers. Max/Last(/) 0/0 rows with 0/0 ms to end.
-> Materialize (cost=0.05..0.17 rows=12 width=8)
Rows out: Avg 1.0 rows x 1 workers. Max/Last(/) 1/1 rows with 0.129/0.129 ms to end, start offset by 0.135/0.135 ms.
-> Append (cost=0.00..0.04 rows=2 width=0)
Rows out: Avg 2.0 rows x 1 workers. Max/Last(/) 2/2 rows with 0.002/0.002 ms to first row, 0.004/0.004 ms to end, start offset by 0.255/0.255 ms.
-> Result (cost=0.00..0.01 rows=1 width=0)
Rows out: Avg 1.0 rows x 1 workers. Max/Last(/) 1/1 rows with 0.002/0.002 ms to end, start offset by 0.255/0.255 ms.
-> Result (cost=0.00..0.01 rows=1 width=0)
Rows out: Avg 1.0 rows x 1 workers. Max/Last(/) 1/1 rows with 0/0 ms to end, start offset by 0.261/0.261 ms.
Slice statistics:
(slice0) Executor memory: 61K bytes.
Statement statistics:
Memory used: 128000K bytes
Settings: default_hash_table_bucket_number=6; optimizer=off
Optimizer status: legacy query optimizer
Data locality statistics:
no data locality information in this query
Total runtime: 0.372 ms
(26 rows)
2) 带有优化器的 hawq 2.0 (orca)
show optimizer;
optimizer
-----------
on
(1 row)
WITH t1 AS ( SELECT 1 c1, 2 c2 UNION ALL SELECT 3 c1, 4 c2 ),
t2 AS ( SELECT 3 c3 )
SELECT * FROM t1
WHERE EXISTS (SELECT * FROM t2 WHERE c3=c1);
c1 | c2
----+----
(0 rows)
EXPLAIN ANALYZE
WITH t1 AS ( SELECT 1 c1, 2 c2 UNION ALL SELECT 3 c1, 4 c2 ),
t2 AS ( SELECT 3 c3 )
SELECT * FROM t1
WHERE EXISTS (SELECT * FROM t2 WHERE c3=c1);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
Nested Loop (cost=0.05..0.29 rows=72 width=8)
Rows out: Avg 0.0 rows x 0 workers. Max/Last(/) 0/0 rows with 0.273/0.273 ms to end.
-> Limit (cost=0.00..0.00 rows=1 width=0)
Rows out: Avg 0.0 rows x 0 workers. Max/Last(/) 0/0 rows with 0.003/0.003 ms to end.
-> Subquery Scan t2 (cost=0.00..0.01 rows=6 width=0)
Rows out: Avg 0.0 rows x 0 workers. Max/Last(/) 0/0 rows with 0.002/0.002 ms to end.
-> Result (cost=0.00..0.01 rows=1 width=0)
One-Time Filter: 3 = $0
Rows out: Avg 0.0 rows x 0 workers. Max/Last(/) 0/0 rows with 0.001/0.001 ms to end.
-> Materialize (cost=0.05..0.17 rows=12 width=8)
Rows out: Avg 1.0 rows x 1 workers. Max/Last(/) 1/1 rows with 0.151/0.151 ms to end, start offset by 0.189/0.189 ms.
-> Append (cost=0.00..0.04 rows=2 width=0)
Rows out: Avg 2.0 rows x 1 workers. Max/Last(/) 2/2 rows with 0.003/0.003 ms to first row, 0.004/0.004 ms to end, start offset by 0.327/0.327 ms.
-> Result (cost=0.00..0.01 rows=1 width=0)
Rows out: Avg 1.0 rows x 1 workers. Max/Last(/) 1/1 rows with 0.002/0.002 ms to end, start offset by 0.327/0.327 ms.
-> Result (cost=0.00..0.01 rows=1 width=0)
Rows out: Avg 1.0 rows x 1 workers. Max/Last(/) 1/1 rows with 0/0 ms to end, start offset by 0.337/0.337 ms.
Slice statistics:
(slice0) Executor memory: 61K bytes.
Statement statistics:
Memory used: 128000K bytes
Settings: default_hash_table_bucket_number=6
Optimizer status: legacy query optimizer
Data locality statistics:
no data locality information in this query
Total runtime: 0.468 ms
(26 rows)
3) 关闭优化器的 hawq 1.x（规划器）
show optimizer;
optimizer
-----------
off
(1 row)
WITH t1 AS ( SELECT 1 c1, 2 c2 UNION ALL SELECT 3 c1, 4 c2 ),
t2 AS ( SELECT 3 c3 )
SELECT * FROM t1
WHERE EXISTS (SELECT * FROM t2 WHERE c3=c1);
c1 | c2
----+----
(0 rows)
EXPLAIN ANALYZE
WITH t1 AS ( SELECT 1 c1, 2 c2 UNION ALL SELECT 3 c1, 4 c2 ),
t2 AS ( SELECT 3 c3 )
SELECT * FROM t1
WHERE EXISTS (SELECT * FROM t2 WHERE c3=c1);
QUERY PLAN
--------------------------------------------------------------------------------------------------
Nested Loop (cost=0.00..0.08 rows=4 width=8)
Rows out: 0 rows with 0.220 ms to end, start offset by 0.093 ms.
-> Append (cost=0.00..0.04 rows=2 width=0)
Rows out: 2 rows with 0.001 ms to first row, 0.003 ms to end, start offset by 0.262 ms.
-> Result (cost=0.00..0.01 rows=1 width=0)
Rows out: 1 rows with 0.001 ms to end, start offset by 0.262 ms.
-> Result (cost=0.00..0.01 rows=1 width=0)
Rows out: 1 rows with 0.001 ms to end, start offset by 0.264 ms.
-> Materialize (cost=0.00..0.01 rows=1 width=0)
Rows out: 0 rows with 0.167 ms to end of 3 scans, start offset by 0.265 ms.
-> Limit (cost=0.00..0.00 rows=1 width=0)
Rows out: 0 rows with 0.003 ms to end, start offset by 0.257 ms.
-> Subquery Scan t2 (cost=0.00..0.01 rows=1 width=0)
Rows out: 0 rows with 0.002 ms to end, start offset by 0.258 ms.
-> Result (cost=0.00..0.01 rows=1 width=0)
One-Time Filter: 3 = $0
Rows out: 0 rows with 0.001 ms to end, start offset by 0.258 ms.
Slice statistics:
(slice0) Executor memory: 61K bytes.
Statement statistics:
Memory used: 128000K bytes
Settings: optimizer=off
Optimizer status: legacy query optimizer
Total runtime: 0.315 ms
(24 rows)
4) 带有优化器的 hawq 1.x (orca)
show optimizer;
optimizer
-----------
on
(1 row)
WITH t1 AS ( SELECT 1 c1, 2 c2 UNION ALL SELECT 3 c1, 4 c2 ),
t2 AS ( SELECT 3 c3 )
SELECT * FROM t1
WHERE EXISTS (SELECT * FROM t2 WHERE c3=c1);
c1 | c2
----+----
3 | 4
(1 row)
EXPLAIN ANALYZE
WITH t1 AS ( SELECT 1 c1, 2 c2 UNION ALL SELECT 3 c1, 4 c2 ),
t2 AS ( SELECT 3 c3 )
SELECT * FROM t1
WHERE EXISTS (SELECT * FROM t2 WHERE c3=c1);
QUERY PLAN
--------------------------------------------------------------------------------------------------------
Hash EXISTS Join (cost=0.00..0.00 rows=2 width=8)
Hash Cond: "outer".c1 = "inner".c3
Rows out: 1 rows with 0.835 ms to first row, 2.373 ms to end, start offset by 0.194 ms.
Executor memory: 1K bytes.
Work_mem used: 1K bytes. Workfile: (0 spilling, 0 reused)
Hash chain length 1.0 avg, 1 max, using 1 of 524341 buckets.
-> Append (cost=0.00..0.00 rows=2 width=8)
Rows out: 2 rows with 0.002 ms to first row, 0.004 ms to end, start offset by 1.023 ms.
-> Result (cost=0.00..0.00 rows=1 width=8)
Rows out: 1 rows with 0.001 ms to first row, 0.002 ms to end, start offset by 1.023 ms.
-> Result (cost=0.00..0.00 rows=1 width=1)
Rows out: 1 rows with 0 ms to end, start offset by 1.024 ms.
-> Result (cost=0.00..0.00 rows=1 width=8)
Rows out: 1 rows with 0.001 ms to end, start offset by 1.026 ms.
-> Result (cost=0.00..0.00 rows=1 width=1)
Rows out: 1 rows with 0 ms to end, start offset by 1.026 ms.
-> Hash (cost=0.00..0.00 rows=1 width=4)
Rows in: 1 rows with 0.010 ms to end, start offset by 1.013 ms.
-> Result (cost=0.00..0.00 rows=1 width=4)
Rows out: 1 rows with 0.006 ms to first row, 0.007 ms to end, start offset by 1.013 ms.
-> Result (cost=0.00..0.00 rows=1 width=1)
Rows out: 1 rows with 0.001 ms to end, start offset by 1.014 ms.
Slice statistics:
(slice0) Executor memory: 8270K bytes. Work_mem: 1K bytes max.
Statement statistics:
Memory used: 128000K bytes
Optimizer status: PQO version 1.591
Total runtime: 2.572 ms
(28 rows)
【讨论】：
以上是关于当优化器参数更改（ON/OFF）时，HAWQ 为相同的 sql 返回不同的输出的主要内容，如果未能解决你的问题，请参考以下文章
SET ANSI_WARNINGS { ON | OFF }