PostgreSQL对or exists产生的filter优化一

Posted robinson1988

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了PostgreSQL对or exists产生的filter优化一相关的知识,希望对你有一定的参考价值。

在Oracle数据库中,where条件中有or exists会走Filter,这种情况一般都需要改写SQL

先创建2个测试表a,b

create table a as select * from dba_objects;
create table b as select * from a;

下面SQL在Oracle中会产生filter,跑得慢

select count(*) from a
where owner='SCOTT' or exists (select null from b where a.object_id=b.data_object_id);

SQL> select count(*) from a
  2  where owner='SCOTT' or exists (select null from b where a.object_id=b.data_object_id);

已用时间:  00: 01: 33.95

执行计划
----------------------------------------------------------
Plan hash value: 1064440332

----------------------------------------------------------------------------
| Id  | Operation           | Name | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------
|   0 | SELECT STATEMENT    |      |     1 |    30 |   290   (1)| 00:00:04 |
|   1 |  SORT AGGREGATE     |      |     1 |    30 |            |          |
|*  2 |   FILTER            |      |       |       |            |          |
|   3 |    TABLE ACCESS FULL| A    | 74940 |  2195K|   290   (1)| 00:00:04 |
|*  4 |    TABLE ACCESS FULL| B    |   800 | 10400 |     3   (0)| 00:00:01 |
----------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter("OWNER"='SCOTT' OR  EXISTS (SELECT 0 FROM "B" "B" WHERE
              "B"."DATA_OBJECT_ID"=:B1))
   4 - filter("B"."DATA_OBJECT_ID"=:B1)

Note
-----
   - dynamic sampling used for this statement (level=2)


统计信息
----------------------------------------------------------
         13  recursive calls
          0  db block gets
   72687576  consistent gets
          0  physical reads
          0  redo size
        424  bytes sent via SQL*Net to client
        416  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed

需要将其改写,秒杀

select count(*)
  from a left join (select data_object_id from b group by data_object_id) b
 on a.object_id = b.data_object_id where a.owner = 'SCOTT' or b.data_object_id is not null;

SQL> select count(*)
  2    from a left join (select data_object_id from b group by data_object_id) b
  3   on a.object_id = b.data_object_id where a.owner = 'SCOTT' or b.data_object_id is not null;

已用时间:  00: 00: 00.02

执行计划
----------------------------------------------------------
Plan hash value: 2430901245

----------------------------------------------------------------------------------------
| Id  | Operation               | Name | Rows  | Bytes |TempSpc| Cost (%CPU)| Time     |
----------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT        |      |     1 |    43 |       |   831   (1)| 00:00:10 |
|   1 |  SORT AGGREGATE         |      |     1 |    43 |       |            |          |
|*  2 |   FILTER                |      |       |       |       |            |          |
|*  3 |    HASH JOIN RIGHT OUTER|      | 80038 |  3360K|  1960K|   831   (1)| 00:00:10 |
|   4 |     VIEW                |      | 80038 |  1016K|       |   294   (2)| 00:00:04 |
|   5 |      HASH GROUP BY      |      | 80038 |  1016K|       |   294   (2)| 00:00:04 |
|   6 |       TABLE ACCESS FULL | B    | 80038 |  1016K|       |   290   (1)| 00:00:04 |
|   7 |     TABLE ACCESS FULL   | A    | 74940 |  2195K|       |   290   (1)| 00:00:04 |
----------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter("A"."OWNER"='SCOTT' OR "B"."DATA_OBJECT_ID" IS NOT NULL)
   3 - access("A"."OBJECT_ID"="B"."DATA_OBJECT_ID"(+))

Note
-----
   - dynamic sampling used for this statement (level=2)


统计信息
----------------------------------------------------------
         28  recursive calls
          0  db block gets
       2714  consistent gets
          0  physical reads
          0  redo size
        424  bytes sent via SQL*Net to client
        416  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed

现在将a,b迁移到PostgreSQL12中

orcl=> select * from version();
                                                 version
---------------------------------------------------------------------------------------------------------
 PostgreSQL 12.7 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
(1 row)

orcl=> show work_mem;
 work_mem
----------
 64MB

orcl=> explain analyze select count(*) from a
where owner='SCOTT' or exists (select null from b where a.object_id=b.data_object_id);
                                                       QUERY PLAN
------------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=158042495.87..158042495.88 rows=1 width=8) (actual time=20.602..20.603 rows=1 loops=1)
   ->  Seq Scan on a  (cost=0.00..158042405.13 rows=36296 width=0) (actual time=8.705..20.384 rows=7144 loops=1)
         Filter: (((owner)::text = 'SCOTT'::text) OR (alternatives: SubPlan 1 or hashed SubPlan 2))
         Rows Removed by Filter: 65441
         SubPlan 1
           ->  Seq Scan on b  (cost=0.00..2177.31 rows=1 width=0) (never executed)
                 Filter: (a.object_id = data_object_id)
         SubPlan 2
           ->  Seq Scan on b b_1  (cost=0.00..1995.85 rows=72585 width=6) (actual time=0.004..5.449 rows=72585 loops=1)
 Planning Time: 0.058 ms
 Execution Time: 20.642 ms
(11 rows)

在PostgreSQL12中,上面SQL秒杀了,而在Oracle中要跑1分33秒

从PG的执行计划中可以看到 alternatives: SubPlan 1 or hashed SubPlan 2 ,意思就是在SubPlan 1 和 hashed SubPlan 2中 二选一

SubPlan 1 有个 never executed,说明没有选择SubPlan 1,选的是SubPlan 2,也就是说没有执行SubPlan 1,执行的是SubPlan 2

SubPlan 2 loops=1,说明只执行了一次

现在调小work_mem,设置为64kB

orcl=> set work_mem='64kB';
SET
orcl=> explain analyze select count(*) from a
where owner='SCOTT' or exists (select null from b where a.object_id=b.data_object_id);
                                                     QUERY PLAN
---------------------------------------------------------------------------------------------------------------------
 Aggregate  (cost=158042495.87..158042495.88 rows=1 width=8) (actual time=222323.048..222323.049 rows=1 loops=1)
   ->  Seq Scan on a  (cost=0.00..158042405.13 rows=36296 width=0) (actual time=4.212..222321.298 rows=7144 loops=1)
         Filter: (((owner)::text = 'SCOTT'::text) OR (SubPlan 1))
         Rows Removed by Filter: 65441
         SubPlan 1
           ->  Seq Scan on b  (cost=0.00..2177.31 rows=1 width=0) (actual time=3.062..3.062 rows=0 loops=72574)
                 Filter: (a.object_id = data_object_id)
                 Rows Removed by Filter: 69980
 Planning Time: 0.057 ms
 Execution Time: 222323.070 ms

将work_mem设置为64kB之后,PG的执行计划其实就和Oracle基本上一模一样了

再来个复杂的 or exists and exists 进行测试

select *
  from a
 where exists (select 1
          from b
         where a.object_id = b.object_id
           and object_type like '%TABLE%')
    or (object_id > 50000 and object_name like 'DBA%')
   and not exists (select 1
          from b
         where a.data_object_id = b.data_object_id
           and object_type like '%INDEX%');

orcl=> explain analyze select *
orcl->   from a
orcl->  where exists (select 1
orcl(>           from b
orcl(>          where a.object_id = b.object_id
orcl(>            and object_type like '%TABLE%')
orcl->     or (object_id > 50000 and object_name like 'DBA%')
orcl->    and not exists (select 1
orcl(>           from b
orcl(>          where a.data_object_id = b.data_object_id
orcl(>            and object_type like '%INDEX%');
                                                                                           QUERY PLAN

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----
 Seq Scan on a  (cost=0.00..342425725.52 rows=36467 width=198) (actual time=5.813..22.910 rows=3285 loops=1)
   Filter: ((alternatives: SubPlan 1 or hashed SubPlan 2) OR ((object_id > '50000'::numeric) AND ((object_name)::text ~~ 'DBA%'::text) AND (NOT (alternatives: SubPlan 3 or hashed SubPlan 4)
)))
   Rows Removed by Filter: 69300
   SubPlan 1
     ->  Seq Scan on b  (cost=0.00..2358.77 rows=1 width=0) (never executed)
           Filter: (((object_type)::text ~~ '%TABLE%'::text) AND (a.object_id = object_id))
   SubPlan 2
     ->  Seq Scan on b b_1  (cost=0.00..2177.31 rows=3075 width=6) (actual time=0.002..5.436 rows=3052 loops=1)
           Filter: ((object_type)::text ~~ '%TABLE%'::text)
           Rows Removed by Filter: 69533
   SubPlan 3
     ->  Seq Scan on b b_2  (cost=0.00..2358.77 rows=1 width=0) (never executed)
           Filter: (((object_type)::text ~~ '%INDEX%'::text) AND (a.data_object_id = data_object_id))
   SubPlan 4
     ->  Seq Scan on b b_3  (cost=0.00..2177.31 rows=4154 width=6) (actual time=0.015..5.368 rows=4260 loops=1)
           Filter: ((object_type)::text ~~ '%INDEX%'::text)
           Rows Removed by Filter: 68325
 Planning Time: 0.099 ms
 Execution Time: 23.021 ms
(19 rows)

从执行计划中可以看到,PG12 也对上面的filter做了优化

结论:当work_mem足够大时,PG会对filter进行优化,将子查询的表进行HASH,让它只扫描一次

           如果work_mem不够大,就不针对filter进行优化

思考:PG能自动优化filter的前提是work_mem足够大,如果exists中表很大怎么办?

          作为DBA,建议不要太依赖这个功能,还是要做好SQL审核(SQL等价改写)才是王道

          

以上是关于PostgreSQL对or exists产生的filter优化一的主要内容,如果未能解决你的问题,请参考以下文章

PostgreSQL对or exists产生的filter优化二

PostgreSQL对or exists产生的filter优化二

PostgreSQL对or exists产生的filter优化一

PostgreSQL对or exists产生的filter优化一

PostgreSQL对or exists的优化

postgresql----IN&&EXISTS