优化在文本搜索中使用正则表达式的 psql 查询
Posted
技术标签:
【中文标题】优化在文本搜索中使用正则表达式的 psql 查询【英文标题】:Optimizing psql query that's using regex on text search 【发布时间】:2022-01-20 00:31:19 【问题描述】:在 psql 中我有以下查询。
关于如何加快/优化它的建议?
我在标题和标题上尝试了各种索引,但都没有被使用。
"SELECT \"people\".* FROM \"people\" WHERE (((TITLE IS NOT NULL AND title ~* '(^| )(one|two|three)( |$|,)' AND title !~* '(^| )(four|five|six)( |$|,)') OR (TITLE IS NULL AND headline ~* '(^| )(one|two|three)( |$|,)' AND headline !~* '(^| )(four|five|six)( |$|,)')) AND ((TITLE IS NOT NULL AND title ~* '(^| )(seven|eight|nine)( |$|,)' AND title !~* '(^| )(ten|eleven)( |$|,)') OR (TITLE IS NULL AND headline ~* '(^| )(seven|eight|nine)( |$|,)' AND headline !~* '(^| )(ten|eleven)( |$|,)')))"
这里是解释:
Gather (cost=1000.00..286343.58 rows=61760 width=715)
Workers Planned: 2
-> Parallel Seq Scan on people (cost=0.00..279167.58 rows=25733 width=715)
Filter: ((((title IS NOT NULL) AND ((title)::text ~* '(^| )(one|two|three)( |$|,)'::text) AND ((title)::text !~* '(^| )(four|five|six)( |$|,)'::text)) OR ((title IS NULL) AND ((headline)::text ~* '(^| )(one|two|three)( |$|,)'::text) AND ((headline)::text !~* '(^| )(four|five|six)( |$|,)'::text))) AND (((title IS NOT NULL) AND ((title)::text ~* '(^| )(seven|eight|nine)( |$|,)'::text) AND ((title)::text !~* '(^| )(ten|eleven)( |$|,)'::text)) OR ((title IS NULL) AND ((headline)::text ~* '(^| )(seven|eight|nine)( |$|,)'::text) AND ((headline)::text !~* '(^| )(ten|eleven)( |$|,)'::text))))
JIT:
Functions: 2
Options: Inlining false, Optimization false, Expressions true, Deforming true
(7 rows)
【问题讨论】:
由于您没有指定列的第一个字符,因此不会使用标准表上的标准索引。尝试使用 postgres 的 full text search 功能。 数据库正则表达式和索引并不是很好。也就是说,实现起来并不是一件简单的事情。我有这个presentation 在这里收藏了一段时间,以满足我过去的需要。这很有趣,可以说明这个问题有多复杂。另一种方法是使用特定模块,例如pg_trgm,尽管它需要对您的查询进行大量调整,祝您好运。抱歉帮不上什么忙 @Bohemian 你把我带到了正确的地方。如果您将其移至答案,我会将其标记为这样。谢谢! 【参考方案1】:除非条件中指定了列的前导部分,否则传统的关系数据库不会对列使用索引,即:
... where my_column like 'FOO%' -- will (usually) use index
... where my_column like '%FOO%' -- will (usually) not use index
要有效地搜索内容中的术语,您需要基于文本的搜索技术。
幸运的是,postgres 提供了对full text search 的支持,这将为您的任务提供出色的性能和方便的语法。
【讨论】:
【参考方案2】:如果我在正确的列上创建一个三元索引,它将直接支持这一点。
令我惊讶的是,它实际上也非常有效。这并不总是给定的,某些正则表达式无法分解为一组有效的三元组。
create extension pg_trgm;
create index on people using gin (title gin_trgm_ops, headline gin_trgm_ops);
给出百万行表的亚毫秒计划:
Bitmap Heap Scan on people (cost=547.25..551.28 rows=1 width=12) (actual time=0.741..0.743 rows=1 loops=1)
Recheck Cond: (((title ~* '(^| )(one|two|three)( |$|,)'::text) OR (headline ~* '(^| )(one|two|three)( |$|,)'::text)) AND ((title ~* '(^| )(seven|eight|nine)( |$|,)'::text) OR (headline ~* '(^| )(seven|eight|nine)( |$|,)'::text)))
Filter: ((((title IS NOT NULL) AND (title ~* '(^| )(one|two|three)( |$|,)'::text) AND (title !~* '(^| )(four|five|six)( |$|,)'::text)) OR ((title IS NULL) AND (headline ~* '(^| )(one|two|three)( |$|,)'::text) AND (headline !~* '(^| )(four|five|six)( |$|,)'::text))) AND (((title IS NOT NULL) AND (title ~* '(^| )(seven|eight|nine)( |$|,)'::text) AND (title !~* '(^| )(ten|eleven)( |$|,)'::text)) OR ((title IS NULL) AND (headline ~* '(^| )(seven|eight|nine)( |$|,)'::text) AND (headline !~* '(^| )(ten|eleven)( |$|,)'::text))))
Rows Removed by Filter: 2
Heap Blocks: exact=1
-> BitmapAnd (cost=547.25..547.25 rows=1 width=0) (actual time=0.701..0.702 rows=0 loops=1)
-> BitmapOr (cost=241.50..241.50 rows=200 width=0) (actual time=0.395..0.395 rows=0 loops=1)
-> Bitmap Index Scan on people_title_headline_idx (cost=0.00..120.75 rows=100 width=0) (actual time=0.208..0.208 rows=80 loops=1)
Index Cond: (title ~* '(^| )(one|two|three)( |$|,)'::text)
-> Bitmap Index Scan on people_title_headline_idx (cost=0.00..120.75 rows=100 width=0) (actual time=0.186..0.186 rows=60 loops=1)
Index Cond: (headline ~* '(^| )(one|two|three)( |$|,)'::text)
-> BitmapOr (cost=305.50..305.50 rows=200 width=0) (actual time=0.301..0.301 rows=0 loops=1)
-> Bitmap Index Scan on people_title_headline_idx (cost=0.00..152.75 rows=100 width=0) (actual time=0.145..0.145 rows=3 loops=1)
Index Cond: (title ~* '(^| )(seven|eight|nine)( |$|,)'::text)
-> Bitmap Index Scan on people_title_headline_idx (cost=0.00..152.75 rows=100 width=0) (actual time=0.156..0.156 rows=2 loops=1)
Index Cond: (headline ~* '(^| )(seven|eight|nine)( |$|,)'::text)
如果没有索引,则需要 500 毫秒。
但是,如果每一行都匹配正表达式,但随后通过匹配负表达式 (!~*
) 排除,则没有索引可以帮助您。
【讨论】:
以上是关于优化在文本搜索中使用正则表达式的 psql 查询的主要内容,如果未能解决你的问题,请参考以下文章