这个查询可以进一步优化吗？即使在较差的硬件上也可以使其快速运行吗？
Posted
技术标签:
【中文标题】这个查询可以进一步优化吗？即使在较差的硬件上也可以使其快速运行吗？【英文标题】：Can this query be further optimized? Can it be made to run quickly even on poor hardware? 【发布时间】：2021-06-05 14:42:34 【问题描述】：我有一个 Spring Boot 应用程序，它使用 Hibernate 生成以下对 postgresql 数据库的查询。在一个快速的本地系统上（6 个带 HT 的快速内核、大量内存、快速 ssd），对数据库的查询以合理的 12-65 毫秒运行。
令我惊讶的是，一旦我部署到 Digital Ocean 的一台虚拟服务器，同一查询的响应下降到令人无法接受的 150-250 毫秒或更高。由于这仍然是一个测试环境，服务器上几乎没有数据或流量，我真的没想到即使是最便宜的服务器，只有 1 个共享 CPU 和 2gb 内存，这个查询会有任何问题。我只是在这里错了吗？将 CPU 升级到更重的东西会使性能恢复到我的预期。
无论如何，下面的查询有什么问题吗？可以进一步优化吗？由于它是应用程序的重要组成部分，我希望它尽可能地高效。由于连接可能会占用大部分执行时间，因此我正在考虑将所有表一起粉碎到一个表中，此时我只需要一个大的 where 子句即可实现相同但没有连接。那会更快吗？
select
mapcellent0_.gameworld_id as gameworl1_10_0_,
mapcellent0_.id as id2_10_0_,
villageent1_.gameworld_id as gameworl1_17_1_,
villageent1_.village_id as village_2_17_1_,
playerenti2_.gameworld_id as gameworl1_13_2_,
playerenti2_.id as id2_13_2_,
kingdoment3_.gameworld_id as gameworl1_8_3_,
kingdoment3_.kingdom_id as kingdom_2_8_3_,
mapcellent0_.kingdom_id as kingdom_3_10_0_,
mapcellent0_.landscape as landscap4_10_0_,
mapcellent0_.oasis as oasis5_10_0_,
mapcellent0_.res_type as res_type6_10_0_,
mapcellent0_.x as x7_10_0_,
mapcellent0_.y as y8_10_0_,
villageent1_.is_city as is_city3_17_1_,
villageent1_.is_main_village as is_main_4_17_1_,
villageent1_.kingdom_id as kingdom_5_17_1_,
villageent1_.map_id as map_id6_17_1_,
villageent1_.name as name7_17_1_,
villageent1_.player_id as player_i8_17_1_,
villageent1_.population as populati9_17_1_,
villageent1_.tribe_id as tribe_i10_17_1_,
villageent1_.x as x11_17_1_,
villageent1_.y as y12_17_1_,
playerenti2_.external_login_token as external3_13_2_,
playerenti2_.kingdom_id as kingdom_4_13_2_,
playerenti2_.name as name5_13_2_,
playerenti2_.role as role6_13_2_,
playerenti2_.treasures as treasure7_13_2_,
playerenti2_.tribe_id as tribe_id8_13_2_,
kingdoment3_.creation_time as creation3_8_3_,
kingdoment3_.kingdom_tag as kingdom_4_8_3_,
kingdoment3_.victory_points as victory_5_8_3_
from
map_cells mapcellent0_
left outer join
villages villageent1_
on mapcellent0_.gameworld_id=villageent1_.gameworld_id
and mapcellent0_.id=villageent1_.map_id
left outer join
players playerenti2_
on villageent1_.gameworld_id=playerenti2_.gameworld_id
and villageent1_.player_id=playerenti2_.id
left outer join
kingdoms kingdoment3_
on playerenti2_.gameworld_id=kingdoment3_.gameworld_id
and playerenti2_.kingdom_id=kingdoment3_.kingdom_id
where
mapcellent0_.gameworld_id=?
and (
mapcellent0_.x in (
? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ?
)
)
and (
mapcellent0_.y in (
? , ? , ? , ? , ? , ? , ? , ? , ? , ? , ?
)
)
order by
mapcellent0_.x asc
在快速机器上对查询运行 EXPLAIN ANALYZE 会产生以下结果：
（请参阅https://explain.depesz.com/s/IYcK 了解图形演示）
"Sort (cost=241.63..242.01 rows=152 width=216) (actual time=1.471..1.475 rows=121 loops=1)"
" Sort Key: mapcellent0_.x"
" Sort Method: quicksort Memory: 47kB"
" -> Hash Left Join (cost=191.63..236.12 rows=152 width=216) (actual time=1.252..1.432 rows=121 loops=1)"
" Hash Cond: ((playerenti2_.gameworld_id = kingdoment3_.gameworld_id) AND (playerenti2_.kingdom_id = kingdoment3_.kingdom_id))"
" -> Hash Right Join (cost=189.69..233.35 rows=152 width=181) (actual time=1.203..1.363 rows=121 loops=1)"
" Hash Cond: ((playerenti2_.gameworld_id = villageent1_.gameworld_id) AND (playerenti2_.id = villageent1_.player_id))"
" -> Seq Scan on players playerenti2_ (cost=0.00..35.34 rows=907 width=77) (actual time=0.019..0.102 rows=907 loops=1)"
" Filter: (gameworld_id = '7fe13a96-f540-4263-9de0-6d20da44fecb'::uuid)"
" -> Hash (cost=187.41..187.41 rows=152 width=104) (actual time=1.153..1.154 rows=121 loops=1)"
" Buckets: 1024 Batches: 1 Memory Usage: 21kB"
" -> Hash Right Join (cost=117.65..187.41 rows=152 width=104) (actual time=0.466..1.107 rows=121 loops=1)"
" Hash Cond: ((villageent1_.gameworld_id = mapcellent0_.gameworld_id) AND (villageent1_.map_id = mapcellent0_.id))"
" -> Seq Scan on villages villageent1_ (cost=0.00..57.41 rows=2353 width=60) (actual time=0.008..0.326 rows=2353 loops=1)"
" Filter: (gameworld_id = '7fe13a96-f540-4263-9de0-6d20da44fecb'::uuid)"
" -> Hash (cost=115.37..115.37 rows=152 width=44) (actual time=0.435..0.435 rows=121 loops=1)"
" Buckets: 1024 Batches: 1 Memory Usage: 17kB"
" -> Index Scan using idx_map_cell_y_only on map_cells mapcellent0_ (cost=0.29..115.37 rows=152 width=44) (actual time=0.035..0.400 rows=121 loops=1)"
" Index Cond: (y = ANY ('-5,-4,-3,-2,-1,0,1,2,3,4,5'::integer[]))"
" Filter: ((gameworld_id = '7fe13a96-f540-4263-9de0-6d20da44fecb'::uuid) AND (x = ANY ('-5,-4,-3,-2,-1,0,1,2,3,4,5'::integer[])))"
" Rows Removed by Filter: 1190"
" -> Hash (cost=1.43..1.43 rows=34 width=35) (actual time=0.028..0.028 rows=34 loops=1)"
" Buckets: 1024 Batches: 1 Memory Usage: 11kB"
" -> Seq Scan on kingdoms kingdoment3_ (cost=0.00..1.43 rows=34 width=35) (actual time=0.005..0.008 rows=34 loops=1)"
" Filter: (gameworld_id = '7fe13a96-f540-4263-9de0-6d20da44fecb'::uuid)"
"Planning Time: 0.600 ms"
"Execution Time: 1.575 ms"
编辑，慢机器的解释（缓冲区，分析）：
Sort (cost=253.87..254.25 rows=152 width=216) (actual time=11.517..11.545 rows=121 loops=1)
Sort Key: mapcellent0_.x
Sort Method: quicksort Memory: 47kB
Buffers: shared hit=23 read=99
I/O Timings: read=7.506
-> Hash Left Join (cost=211.99..248.36 rows=152 width=216) (actual time=10.175..11.374 rows=121 loops=1)
Hash Cond: ((playerenti2_.gameworld_id = kingdoment3_.gameworld_id) AND (playerenti2_.kingdom_id = kingdoment3_.kingdom_id))
Buffers: shared hit=20 read=99
I/O Timings: read=7.506
-> Hash Right Join (cost=210.19..245.73 rows=152 width=181) (actual time=9.684..10.826 rows=121 loops=1)
Hash Cond: ((playerenti2_.gameworld_id = villageent1_.gameworld_id) AND (playerenti2_.id = villageent1_.player_id))
Buffers: shared hit=20 read=98
I/O Timings: read=7.121
-> Seq Scan on players playerenti2_ (cost=0.00..27.64 rows=851 width=77) (actual time=0.377..1.291 rows=850 loops=1)
Filter: (gameworld_id = '2fb2192a-9373-41e9-9119-6e73d2a07154'::uuid)
Buffers: shared read=17
I/O Timings: read=1.073
-> Hash (cost=207.91..207.91 rows=152 width=104) (actual time=9.267..9.281 rows=121 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 21kB
Buffers: shared hit=17 read=81
I/O Timings: read=6.049
-> Hash Right Join (cost=109.65..207.91 rows=152 width=104) (actual time=5.617..9.147 rows=121 loops=1)
Hash Cond: ((villageent1_.gameworld_id = mapcellent0_.gameworld_id) AND (villageent1_.map_id = mapcellent0_.id))
Buffers: shared hit=17 read=81
I/O Timings: read=6.049
-> Seq Scan on villages villageent1_ (cost=0.00..85.76 rows=2381 width=60) (actual time=0.362..2.983 rows=2380 loops=1)
Filter: (gameworld_id = '2fb2192a-9373-41e9-9119-6e73d2a07154'::uuid)
Buffers: shared read=56
I/O Timings: read=2.145
-> Hash (cost=107.37..107.37 rows=152 width=44) (actual time=5.176..5.188 rows=121 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 17kB
Buffers: shared hit=17 read=25
I/O Timings: read=3.904
-> Index Scan using idx_map_cell_y_only on map_cells mapcellent0_ (cost=0.29..107.37 rows=152 width=44) (actual time=1.403..5.054 rows=121 loops=1)
Index Cond: (y = ANY ('-5,-4,-3,-2,-1,0,1,2,3,4,5'::integer[]))
Filter: ((gameworld_id = '2fb2192a-9373-41e9-9119-6e73d2a07154'::uuid) AND (x = ANY ('-5,-4,-3,-2,-1,0,1,2,3,4,5'::integer[])))
Rows Removed by Filter: 1190
Buffers: shared hit=17 read=25
I/O Timings: read=3.904
-> Hash (cost=1.36..1.36 rows=29 width=35) (actual time=0.446..0.447 rows=29 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
Buffers: shared read=1
I/O Timings: read=0.385
-> Seq Scan on kingdoms kingdoment3_ (cost=0.00..1.36 rows=29 width=35) (actual time=0.402..0.415 rows=29 loops=1)
Filter: (gameworld_id = '2fb2192a-9373-41e9-9119-6e73d2a07154'::uuid)
Buffers: shared read=1
I/O Timings: read=0.385
Planning Time: 15.927 ms
Execution Time: 11.724 ms
(49 rows)
https://explain.depesz.com/s/1eEb
再次运行它给了我：
Sort (cost=253.87..254.25 rows=152 width=216) (actual time=2.174..2.187 rows=121 loops=1)
Sort Key: mapcellent0_.x
Sort Method: quicksort Memory: 47kB
Buffers: shared hit=113
-> Hash Left Join (cost=211.99..248.36 rows=152 width=216) (actual time=1.644..2.075 rows=121 loops=1)
Hash Cond: ((playerenti2_.gameworld_id = kingdoment3_.gameworld_id) AND (playerenti2_.kingdom_id = kingdoment3_.kingdom_id))
Buffers: shared hit=113
-> Hash Right Join (cost=210.19..245.73 rows=152 width=181) (actual time=1.614..1.983 rows=121 loops=1)
Hash Cond: ((playerenti2_.gameworld_id = villageent1_.gameworld_id) AND (playerenti2_.id = villageent1_.player_id))
Buffers: shared hit=112
-> Seq Scan on players playerenti2_ (cost=0.00..27.64 rows=851 width=77) (actual time=0.005..0.193 rows=850 loops=1)
Filter: (gameworld_id = '2fb2192a-9373-41e9-9119-6e73d2a07154'::uuid)
Buffers: shared hit=17
-> Hash (cost=207.91..207.91 rows=152 width=104) (actual time=1.593..1.594 rows=121 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 21kB
Buffers: shared hit=95
-> Hash Right Join (cost=109.65..207.91 rows=152 width=104) (actual time=0.630..1.545 rows=121 loops=1)
Hash Cond: ((villageent1_.gameworld_id = mapcellent0_.gameworld_id) AND (villageent1_.map_id = mapcellent0_.id))
Buffers: shared hit=95
-> Seq Scan on villages villageent1_ (cost=0.00..85.76 rows=2381 width=60) (actual time=0.009..0.490 rows=2380 loops=1)
Filter: (gameworld_id = '2fb2192a-9373-41e9-9119-6e73d2a07154'::uuid)
Buffers: shared hit=56
-> Hash (cost=107.37..107.37 rows=152 width=44) (actual time=0.602..0.603 rows=121 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 17kB
Buffers: shared hit=39
-> Index Scan using idx_map_cell_y_only on map_cells mapcellent0_ (cost=0.29..107.37 rows=152 width=44) (actual time=0.047..0.558 rows=121 loops=1)
Index Cond: (y = ANY ('-5,-4,-3,-2,-1,0,1,2,3,4,5'::integer[]))
Filter: ((gameworld_id = '2fb2192a-9373-41e9-9119-6e73d2a07154'::uuid) AND (x = ANY ('-5,-4,-3,-2,-1,0,1,2,3,4,5'::integer[])))
Rows Removed by Filter: 1190
Buffers: shared hit=39
-> Hash (cost=1.36..1.36 rows=29 width=35) (actual time=0.020..0.020 rows=29 loops=1)
Buckets: 1024 Batches: 1 Memory Usage: 10kB
Buffers: shared hit=1
-> Seq Scan on kingdoms kingdoment3_ (cost=0.00..1.36 rows=29 width=35) (actual time=0.004..0.011 rows=29 loops=1)
Filter: (gameworld_id = '2fb2192a-9373-41e9-9119-6e73d2a07154'::uuid)
Buffers: shared hit=1
Planning Time: 1.374 ms
Execution Time: 2.309 ms
(38 rows)
编辑 2：
直接在数据库本身上执行这些查询实际上并没有那么慢。我认为我过于专注于优化查询本身。
我正在向我的（慢速）服务器发送请求，这可能需要 80 到 300 毫秒才能完成。基本上，我在一个发送地图数据请求的站点上滚动，它还没有在本地 javascript 存储中。滚动后，从同一位置开始，我希望 postgres 的缓存会被预热。
生成要发送到服务器的 SQL 的部分代码正在通过一个简单的 println 语句进行测量：
val pointsFromDbInMs = measureTimeMillis
mapCells = mapRepository.findByIdGameworldIdAndXInAndYInOrderByX(gameworldId,
points.map it.x .distinct(),
points.map it.y .distinct())
println("Got $mapCells.size points from database in: $pointsFromDbInMs ms")
在此之前我测量了 REST 控制器的所有内容，之后测量了所有代码。这是唯一需要任何不可忽略的毫秒才能完成的事情。因此，如果不是数据库本身需要很长时间才能完成查询，那么它是 Hibernate/Spring boot 和数据库之间的任何东西......？或者仅仅是糟糕的单核处理器无法足够快地将 postgres 记录转换为 Java/Kotlin 对象..？
【问题讨论】：
你能不能打开track_io_timing（不知道DO会不会让你），然后把EXPLAIN（ANALYZE，BUFFERS）收集到慢机器上，而不是快机器上？ 看起来不错，可以让我打开它！为慢速机器添加了解释（分析，缓冲区）。 查询在“慢”机器上运行 11 毫秒，而不是 200 毫秒。 您是否多次运行查询并忽略初始结果以允许数据库预热其缓存？ 【参考方案1】：如果您想知道时间花在了哪里，请使用像 Async Profiler 这样的分析器：https://github.com/jvm-profiling-tools/async-profiler
IntelliJ 有一个很好的集成，但您也可以自己运行它：https://www.jetbrains.com/help/idea/async-profiler.html
【讨论】：
以上是关于这个查询可以进一步优化吗？即使在较差的硬件上也可以使其快速运行吗？的主要内容，如果未能解决你的问题，请参考以下文章