需要 POSTGRES 调优的建议

Posted 2023-04-15

【中文标题】需要 POSTGRES 调优的建议

【英文标题】：Need suggestion for POSTGRES Tuning

【发布时间】：2020-07-21 16:32:49

【问题描述】：

我需要在我的数据库上经常运行几个复杂的查询，我的一个查询需要大约 57 秒才能执行，我正在尝试一种调整我的 Postgres 数据库的方法，所以我可以缩短执行时间到一半或更多。

我已经在表格上创建了​​索引并调整了一些参数，但对我没有帮助。

我使用的是 Postgres 9.6 版，以下是我用来调整数据库的参数。

max_connections = 1500 shared_buffers =8GB 有效缓存大小 = 8GB maintenance_work_mem = 2GB checkpoint_completion_target = 0.7 wal_buffers = 16MB default_statistics_target = 100 random_page_cost = 1.1 effective_io_concurrency = 200 work_mem = 910kB min_wal_size = 1GB max_wal_size = 4GB max_worker_processes = 48 max_parallel_workers_per_gather = 24

查询如下

Explain Analyze SELECT a. SD_ID, a.SIP_PART, a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID,
TO_CHAR(MIN(a.SIPD_DATE), 'YYYY/MM/DD HH24:MI:SS'), SUM(CASE WHEN a.STATUS < 30 THEN a.THEO_SECONDS ELSE 0 END ), MAX(a.TRAIL_SECONDS), x. SD_ID, x.SIP_PART,
x.plan_notes, x.STEP_NO, x.PRODUCT_TYPE, x.P_ID,  TO_CHAR(MIN(x.SIPD_DATE), 'YYYY/MM/DD HH24:MI:SS'),
 SUM(x.THEO_SECONDS), SUM(CASE WHEN x.STATUS >= 30 THEN x.THEO_SECONDS ELSE 0 END ), MAX(x.TRAIL_SECONDS),
 MIN(a.PSEQ - x.PSEQ), CASE WHEN x. SD_ID <> a. SD_ID THEN MIN(x.BSEQ - a.BSEQ) ELSE 0 END, MIN(x.PSEQ), '', 0, 0, COUNT(1),
 SUM(CASE WHEN x.STATUS >= 30 THEN 1 ELSE 0 END ) , CASE WHEN MIN(x.BSEQ) > 0 AND MIN(a.BSEQ) > 0 AND MIN(x.BSEQ) <> MIN(a.BSEQ) AND MIN(x.POS) = MIN(a.POS) THEN MIN(x.STEP - a.STEP) ELSE 0 END
 FROM MYTESTABLE a, MYTESTABLE x
 WHERE a.SIPD_DATE >=  TO_TIMESTAMP('2020/07/20 00:00:00','YYYY/MM/DD HH24:MI:SS')
 AND a.SIPD_DATE <=  TO_TIMESTAMP('2020/07/20 23:59:59','YYYY/MM/DD HH24:MI:SS')  AND  a.HISTORY = 0 AND a.PSEQ > 0  AND x.HISTORY = 0 AND x.PSEQ > 0
 AND a.ORDER_NO = x.ORDER_NO  AND a.ITEM_NO = x.ITEM_NO  AND a.QTY = x.QTY  AND (       (a. SD_ID > 0 AND x. SD_ID > 0 AND a. SD_ID <> x. SD_ID
 AND ((a.BSEQ = 1 AND x.BSEQ > 1) OR (a.BSEQ > 1 AND x.BSEQ = 1)))
 OR       (a. SD_ID = x. SD_ID AND ((a.BSEQ = 1 AND x.BSEQ > 1) OR (a.BSEQ > 1 AND x.BSEQ = 1)))
 OR       (a. SD_ID = x. SD_ID AND a.BSEQ = x.BSEQ AND         ((a.PRODUCT_TYPE IS NULL AND x.PRODUCT_TYPE IS NULL)
 OR (a.PRODUCT_TYPE IS NOT NULL AND a.PRODUCT_TYPE = x.PRODUCT_TYPE AND a.P_ID = x.P_ID))
 AND         (a.plan_notes <> a.plan_notes OR a.STEP_NO <> x.STEP_NO) AND         a.PSEQ <> x.PSEQ)
 OR       (a. SD_ID = x. SD_ID AND a.BSEQ > 1 AND x.BSEQ > 1 AND a.BSEQ <> x.BSEQ AND        a.PRODUCT_TYPE = 1 AND x.PRODUCT_TYPE = 1
 AND a.P_ID <> x.P_ID AND a.POS = x.POS AND       ((a.STEP = 0 AND x.STEP > 0) OR (x.STEP = 0 AND a.STEP > 0))))
 GROUP BY a. SD_ID, a.SIP_PART,  a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID,  DATE_TRUNC('day', a.SIPD_DATE) ,
 x. SD_ID, x.SIP_PART,  x.plan_notes, x.STEP_NO, x.PRODUCT_TYPE, x.P_ID,  DATE_TRUNC('day', x.SIPD_DATE)  ORDER BY a. SD_ID,
 a.SIP_PART,  a.plan_notes, a.STEP_NO, a.PRODUCT_TYPE, a.P_ID,  DATE_TRUNC('day', a.SIPD_DATE) , x. SD_ID, x.SIP_PART,  MIN(x.PSEQ);

下面是查询计划

                                                                     QUERY PLAN


 Sort  (cost=5426751.71..5436426.08 rows=3869747 width=482) (actual time=57121.003..57139.244 rows=105225 loops=1)
   Sort Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, (min(x.PSEQ))
   Sort Method: external sort  Disk: 19848kB
   ->  GroupAggregate  (cost=1982498.13..2524262.71 rows=3869747 width=482) (actual time=34888.529..56516.255 rows=105225 loops=1)
         Group Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, x.plan_notes, x.step_no, x.PRODUCT_TYPE, x.P_ID, (dat
e_trunc('day'::text, x.SIPd_date))
         ->  Sort  (cost=1982498.13..1992172.50 rows=3869747 width=146) (actual time=34888.391..46834.281 rows=3195855 loops=1)
               Sort Key: a. SD_ID, a.SIP_part, a.plan_notes, a.step_no, a.PRODUCT_TYPE, a.P_ID, (date_trunc('day'::text, a.SIPd_date)), x. SD_ID, x.SIP_part, x.plan_notes, x.step_no, x.PRODUCT_TYPE, x.P_ID,
 (date_trunc('day'::text, x.SIPd_date))
               Sort Method: external merge  Disk: 519008kB
               ->  Gather  (cost=1000.56..877326.14 rows=3869747 width=146) (actual time=10.146..1648.706 rows=3195855 loops=1)
                     Workers Planned: 6
                     Workers Launched: 6
                     ->  Nested Loop  (cost=0.56..473227.49 rows=644958 width=146) (actual time=10.876..5330.931 rows=456551 loops=7)
                           ->  Parallel Seq Scan on MYTESTABLE a  (cost=0.00..278565.05 rows=167365 width=82) (actual time=10.771..975.060 rows=79290 loops=7)
                                 Filter: ((PSEQ > '0'::numeric) AND (history = '0'::numeric) AND (SIPd_date >= to_timestamp('2020/07/20 00:00:00'::text, 'YYYY/MM/DD HH24:MI:SS'::text)) AND (SIPd_date <= to_timestamp('2020/0
7/20 23:59:59'::text, 'YYYY/MM/DD HH24:MI:SS'::text)))
                                 Rows Removed by Filter: 386673
                           ->  Index Scan using idx_a1 on MYTESTABLE x  (cost=0.56..1.13 rows=1 width=82) (actual time=0.020..0.049 rows=6 loops=555028)
                                 Index Cond: ((order_no = a.order_no) AND (item_no = a.item_no) AND (QTY = a.QTY) AND (history = '0'::numeric) AND (PSEQ > '0'::numeric))
                                 Filter: (((a. SD_ID > '0'::numeric) AND ( SD_ID > '0'::numeric) AND (a. SD_ID <>  SD_ID) AND (((a.BSEQ = '1'::numeric) AND (BSEQ > '1'::numeric)) OR ((a.BSEQ > '1'::nume
ric) AND (BSEQ = '1'::numeric)))) OR ((a. SD_ID =  SD_ID) AND (((a.BSEQ = '1'::numeric) AND (BSEQ > '1'::numeric)) OR ((a.BSEQ > '1'::numeric) AND (BSEQ = '1'::numeric)))) OR ((a. SD_ID = SIP_
id) AND (a.BSEQ = BSEQ) AND (((a.PRODUCT_TYPE IS NULL) AND (PRODUCT_TYPE IS NULL)) OR ((a.PRODUCT_TYPE IS NOT NULL) AND (a.PRODUCT_TYPE = PRODUCT_TYPE) AND (a.P_ID = P_ID))) AND ((a.plan_notes <> a.plan_notes) OR (a.step_no <> step_n
o)) AND (a.PSEQ <> PSEQ)) OR ((a. SD_ID =  SD_ID) AND (a.BSEQ > '1'::numeric) AND (BSEQ > '1'::numeric) AND (a.BSEQ <> BSEQ) AND (a.PRODUCT_TYPE = '1'::numeric) AND (PRODUCT_TYPE = '1'::numeric) AND (a.mat_i
d <> P_ID) AND (a.POS = POS) AND (((a.STEP = '0'::numeric) AND (STEP > '0'::numeric)) OR ((STEP = '0'::numeric) AND (a.STEP > '0'::numeric)))))
                                 Rows Removed by Filter: 3
 Planning time: 9.241 ms
 Execution time: 57280.673 ms

【问题讨论】：

您对 max_connections 和 work_mem 的设置似乎相当极端（分别为高和低）。这些设置背后有什么原因吗？

是的，我的数据库与 4 个应用程序连接，这就是我将 max_connection 设置为 1500 的原因，对于 work_mem，我已经尝试增加它但没有运气，我从pgtune.leopard.in.ua。我已经添加到测试中。

1500/4 = 375，这仍然很高，可能您的应用程序正在泄漏连接，或者只需要使用连接池。你尝试了多高的work_mem？计划没有改变，还是改变了但没有变得更快？您可以将其设置为本地到会话以进行测试，因此您应该能够将其设置得相当高。

你测试过升级吗？特别是，最大的排序应该在较新的版本中并行完成。并行执行是 9.6 中的新功能，非常原始。

【参考方案1】：

更新到postgres10后修复了，jjanes的建议奏效了