数据分布不均衡导致性能问题
Posted robinson1988
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了数据分布不均衡导致性能问题相关的知识，希望对你有一定的参考价值。
今晚(2016/04/14)数据库版本11.2.0.4 遇到一个奇葩案例，虽然之前也遇到过非常多奇葩案例，
但是限于当时条件，无法收集案例，谁叫他奶奶的银行，证券，电信不允许泄密啊。还好今晚这个案例可以拿出来分享。
故事是这样的，下面这个SQL要跑几十分钟
select count(distinct a.user_name), count(distinct a.invest_id)
from base_data_login_info@agent a
where a.str_day <= '20160304'
and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002)
and a.platform = a.platform;
Plan hash value: 2367445948
-------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Inst |IN-OUT|
-------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 130 | 754 (2)| 00:00:10 | | |
| 1 | SORT GROUP BY | | 1 | 130 | | | | |
|* 2 | HASH JOIN | | 4067K| 504M| 754 (2)| 00:00:10 | | |
|* 3 | HASH JOIN | | 11535 | 360K| 258 (1)| 00:00:04 | | |
|* 4 | TABLE ACCESS FULL| TB_USER_CHANNEL | 11535 | 157K| 19 (0)| 00:00:01 | | |
| 5 | TABLE ACCESS FULL| TB_CHANNEL_INFO | 11767 | 206K| 238 (0)| 00:00:03 | | |
| 6 | REMOTE | BASE_DATA_LOGIN_INFO | 190K| 17M| 486 (1)| 00:00:06 | AGENT | R->S |
-------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("A"."CHANNEL_ID"="CHANNEL_RLAT")
3 - access("A"."CHANNEL_ID"="B"."CHANNEL_ID")
4 - filter("A"."USER_ID"=5002)
Remote SQL Information (identified by operation id):
----------------------------------------------------
6 - SELECT "USER_NAME","INVEST_ID","STR_DAY","CHANNEL_ID","PLATFORM" FROM "BASE_DATA_LOGIN_INFO"
"A" WHERE "STR_DAY"<='20160304' AND "STR_DAY">='20160301' AND "PLATFORM" IS NOT NULL (accessing
'AGENT' )
我瞄了一眼执行计划，初步一看执行计划正常啊。然后赶紧问问dblink的表有多大， in 里面 a, b 分别有多大
tb_user_channel 1W
tb_channel_info 1W
base_data_login_info 19W 过滤剩下4w
这些表都不大，最大一个才19w行，怎么也不可能跑几十分钟啊。然后我开始怀疑是不是dblink的表产生了性能问题。
为了排除dblink的表产生性能问题，我让哥们在本地创建一个一模一样的表，结果还是慢，速度根本没有一丁点改变。
大爷的，老虎不发威，当我病猫啊。之前都是瞄一眼搞定一个SQL优化。大爷的这次栽了。
以我优化了几万个SQL的功力，恩这个鸟SQL我得花1分钟搞定它。
于是我让哥们跑下面这个SQL
select count(*)
from base_data_login_info@agent a
where a.str_day <= '20160304'
and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002)
and a.platform = a.platform;
秒杀，没看错，是秒杀 大爷的 奇怪了，这SQL居然秒杀了。 然后我再让哥们跑下面这个SQL
select count(a.user_name)
from base_data_login_info@agent a
where a.str_day <= '20160304'
and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002)
and a.platform = a.platform;
秒杀，于是再让哥们跑下面SQL
select count(a.user_name), count(a.invest_id)
from base_data_login_info@agent a
where a.str_day <= '20160304'
and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002)
and a.platform = a.platform;
秒杀，你大爷的，再跑一下下面这个SQL
select count(distinct a.user_name), count(a.invest_id)
from base_data_login_info@agent a
where a.str_day <= '20160304'
and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002)
and a.platform = a.platform;
又秒杀了，卧槽，我感觉女神就在我面前了，我再加一个distinct看看还能不能秒杀
select count(distinct a.user_name), count(distinct a.invest_id)
from base_data_login_info@agent a
where a.str_day <= '20160304'
and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002)
and a.platform = a.platform;
这次死了，SQL跑不动了，太他妈奇葩了，看文章的兄弟们，你们觉得是不是很奇葩。说了这么多，遇到这种奇葩的问题怎么解决呢？
首先要解决问题啊，不能让这个SQL跑得慢，搞不定问题，那哥也不用混了，道森也不用开了，倒闭得了。
其次嘛再找出根本问题，防止下一次遇到同类问题，顺便也让网友看看我写的案例，各位网友就当黄色小说看看得了。
先来解决这个问题，给了兄弟下面这个SQL
with t1 as
(select /*+ materialize */
a.user_name, a.invest_id
from base_data_login_info@agent a
where a.str_day <= '20160304' and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat from tb_user_channel a, tb_channel_info b where a.channel_id = b.channel_id and a.user_id = 5002)
and a.platform = a.platform)
select count(distinct user_name) ,count(distinct invest_id) from t1;
Plan hash value: 901326807
-----------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Inst |IN-OUT|
-----------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 54 | 1621 (1)| 00:00:20 | | |
| 1 | TEMP TABLE TRANSFORMATION | | | | | | | |
| 2 | LOAD AS SELECT | SYS_TEMP_0FD9D6720_EB8EA | | | | | | |
|* 3 | HASH JOIN RIGHT SEMI | | 190K| 22M| 744 (1)| 00:00:09 | | |
| 4 | VIEW | VW_NSO_1 | 11535 | 304K| 258 (1)| 00:00:04 | | |
|* 5 | HASH JOIN | | 11535 | 360K| 258 (1)| 00:00:04 | | |
|* 6 | TABLE ACCESS FULL | TB_USER_CHANNEL | 11535 | 157K| 19 (0)| 00:00:01 | | |
| 7 | TABLE ACCESS FULL | TB_CHANNEL_INFO | 11767 | 206K| 238 (0)| 00:00:03 | | |
| 8 | REMOTE | BASE_DATA_LOGIN_INFO | 190K| 17M| 486 (1)| 00:00:06 | AGENT | R->S |
| 9 | SORT GROUP BY | | 1 | 54 | | | | |
| 10 | VIEW | | 190K| 9M| 878 (1)| 00:00:11 | | |
| 11 | TABLE ACCESS FULL | SYS_TEMP_0FD9D6720_EB8EA | 190K| 9M| 878 (1)| 00:00:11 | | |
-----------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
3 - access("A"."CHANNEL_ID"="CHANNEL_RLAT")
5 - access("A"."CHANNEL_ID"="B"."CHANNEL_ID")
6 - filter("A"."USER_ID"=5002)
Remote SQL Information (identified by operation id):
----------------------------------------------------
8 - SELECT "USER_NAME","INVEST_ID","STR_DAY","CHANNEL_ID","PLATFORM" FROM "BASE_DATA_LOGIN_INFO" "A" WHERE
"STR_DAY"<='20160304' AND "STR_DAY">='20160301' AND "PLATFORM" IS NOT NULL (accessing 'AGENT' )
SQL秒杀了。 with as /*+ materialize */ 这个绝招 道森的人都知道。
不信你看我博客去啊(百度 csdn 落落的专栏)。我估计过不了多久整个 数据库圈的人全都知道了。
光解决问题，那不行啊，必须找出问题根本原因啊，这样才好装逼装大神装大师嘛。
首先从执行计划上分析
跑得快的SQL以及执行计划
select count(a.user_name), count(distinct a.invest_id)
from base_data_login_info@agent a
where a.str_day <= '20160304'
and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002)
and a.platform = a.platform
Plan hash value: 4282421321
------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes |TempSpc| Cost (%CPU)| Time | Inst |IN-OUT|
------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 40 | | 2982 (1)| 00:00:36 | | |
| 1 | SORT AGGREGATE | | 1 | 40 | | | | | |
| 2 | VIEW | VW_DAG_0 | 41456 | 1619K| | 2982 (1)| 00:00:36 | | |
| 3 | HASH GROUP BY | | 41456 | 4250K| 20M| 2982 (1)| 00:00:36 | | |
|* 4 | HASH JOIN RIGHT SEMI| | 190K| 19M| | 744 (1)| 00:00:09 | | |
| 5 | VIEW | VW_NSO_1 | 11535 | 80745 | | 258 (1)| 00:00:04 | | |
|* 6 | HASH JOIN | | 11535 | 360K| | 258 (1)| 00:00:04 | | |
|* 7 | TABLE ACCESS FULL| TB_USER_CHANNEL | 11535 | 157K| | 19 (0)| 00:00:01 | | |
| 8 | TABLE ACCESS FULL| TB_CHANNEL_INFO | 11767 | 206K| | 238 (0)| 00:00:03 | | |
| 9 | REMOTE | BASE_DATA_LOGIN_INFO | 190K| 17M| | 486 (1)| 00:00:06 | AGENT | R->S |
------------------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
4 - access("A"."CHANNEL_ID"="CHANNEL_RLAT")
6 - access("A"."CHANNEL_ID"="B"."CHANNEL_ID")
7 - filter("A"."USER_ID"=5002)
Remote SQL Information (identified by operation id):
----------------------------------------------------
9 - SELECT "USER_NAME","INVEST_ID","STR_DAY","CHANNEL_ID","PLATFORM" FROM "BASE_DATA_LOGIN_INFO" "A" WHERE
"STR_DAY"<='20160304' AND "STR_DAY">='20160301' AND "PLATFORM" IS NOT NULL (accessing 'AGENT' )
跑得慢的SQL以及执行计划
select count(distinct a.user_name), count(distinct a.invest_id)
from base_data_login_info@agent a
where a.str_day <= '20160304'
and a.str_day >= '20160301'
and a.channel_id in (select channel_rlat
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002)
and a.platform = a.platform
Plan hash value: 2367445948
-------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | Inst |IN-OUT|
-------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 130 | 754 (2)| 00:00:10 | | |
| 1 | SORT GROUP BY | | 1 | 130 | | | | |
|* 2 | HASH JOIN | | 4067K| 504M| 754 (2)| 00:00:10 | | |
|* 3 | HASH JOIN | | 11535 | 360K| 258 (1)| 00:00:04 | | |
|* 4 | TABLE ACCESS FULL| TB_USER_CHANNEL | 11535 | 157K| 19 (0)| 00:00:01 | | |
| 5 | TABLE ACCESS FULL| TB_CHANNEL_INFO | 11767 | 206K| 238 (0)| 00:00:03 | | |
| 6 | REMOTE | BASE_DATA_LOGIN_INFO | 190K| 17M| 486 (1)| 00:00:06 | AGENT | R->S |
-------------------------------------------------------------------------------------------------------------
Predicate Information (identified by operation id):
---------------------------------------------------
2 - access("A"."CHANNEL_ID"="CHANNEL_RLAT")
3 - access("A"."CHANNEL_ID"="B"."CHANNEL_ID")
4 - filter("A"."USER_ID"=5002)
Remote SQL Information (identified by operation id):
----------------------------------------------------
6 - SELECT "USER_NAME","INVEST_ID","STR_DAY","CHANNEL_ID","PLATFORM" FROM "BASE_DATA_LOGIN_INFO"
"A" WHERE "STR_DAY"<='20160304' AND "STR_DAY">='20160301' AND "PLATFORM" IS NOT NULL (accessing
'AGENT' )
如果没有优化过几千几万个SQL，哪里能练出火眼金睛，注意看跑得慢的SQL是HASH JOIN，跑得快的SQL是 HASH JOIN RIGHT SEMI
也就是说跑得慢的SQL是 HASH JOIN(inner join)，跑得快的 SQL 是 HASH SEMI JOIN (semi join)
说人话就是跑得慢的SQL变成内连接了，跑得快的SQL是半连接(in/exists)。
明明SQL是半连接啊，咋变成内连接了呢，这涉及到优化器内部原理和大学课程里面的关系代数了这里就不装逼了，免得到时候一个个看不懂来问我烦死了。
问题又来了，就几万跟十几万的进行HASH JOIN 应该很快啊，如果跑的慢那只有一个解释，2个表的关联列数据分布都非常不均衡
19W 表连接列
SQL> select channel_id,count(*) from base_data_login_info group by channel_id order by 2;
CHANNEL_ID COUNT(*)
-------------------------------------------------- ----------
011a1 2
003a1 3
021a1 3
006a1 12
024h2 16
013a1 19
007a1 24
012a1 25
005a1 27
EPT01 36
028h2 109
008a1 139
029a1 841
009a1 921
014a1 1583
000a1 1975
a0001 2724
004a1 5482
001a1 16329
026h2 160162
in里面的关联列数据分布
select channel_rlat, count(*)
from tb_user_channel a, tb_channel_info b
where a.channel_id = b.channel_id
and a.user_id = 5002
group by channel_rlat
order by 2 desc
channel_rlat count(*)
026h2 10984
024h2 7
002h2 6
023a2 2
007s001022001 1
007s001022002 1
007s001024007 1
007s001024009 1
007s001022009 1
001s001006 1
001s001008 1
001s001001001 1
001s001001003 1
001s001001007 1
001s001001014 1
007s001018003 1
007s001018007 1
007s001019005 1
007s001019008 1
001s001002011 1
007s001011003 1
007s001034 1
007s001023005 1
果然，不出本大仙所料，这尼玛走内连接的 HASH JOIN 不死人才怪
026h2 160162 与 026h2 10984 进行关联完全就是一个笛卡尔积
10046 trace 文件已经 告诉了答案 HASH JOIN 返回 410996039 ，这尼玛就是一个小型笛卡尔积了
Rows (1st) Rows (avg) Rows (max) Row Source Operation
---------- ---------- ---------- ---------------------------------------------------
1 1 1 SORT GROUP BY (cr=3643 pr=0 pw=0 time=1236559678 us)
410996039 410996039 410996039 HASH JOIN (cr=3643 pr=0 pw=0 time=406365130 us cost=1006 size=66968010 card=458685)
11535 11535 11535 HASH JOIN (cr=945 pr=0 pw=0 time=199182 us cost=258 size=369120 card=11535)
11535 11535 11535 TABLE ACCESS FULL TB_USER_CHANNEL (cr=67 pr=0 pw=0 time=21452 us cost=19 size=161490 card=11535)
11771 11771 11771 TABLE ACCESS FULL TB_CHANNEL_INFO (cr=878 pr=0 pw=0 time=30291 us cost=238 size=211806 card=11767)
45122 45122 45122 TABLE ACCESS FULL BASE_DATA_LOGIN_INFO (cr=2698 pr=0 pw=0 time=218144 us cost=747 size=2447922 card=21473)
看不懂的人可以做个实验
create table a as select * from dba_objects;
create table b as select * from dba_objects;
然后你去跑下面的SQL，慢慢等结果把
select count(distinct owner), count(distinct object_name)
from a
where owner in (select owner from b);
然而你跑下面这些SQL都可以秒杀
select count(owner), count(distinct object_name)
from a
where owner in (select owner from b);
select count(distinct owner), count(distinct object_name)
from a
where object_id in (select object_id from b);
那么怎么对跑得慢的SQL进行等价改写呢？
select count(distinct owner), count(distinct object_name)
from a
where owner in (select owner from b);
答案如下：
select count(distinct owner), count(distinct object_name)
from (select owner, object_name
from a
where owner in (select owner from b)
and rownum > 0);思考为啥11g CBO会 改写为 inner join 呢？ select xxx from 1的表 where owner in (select owner from n 的表) 改写为 inner join 前面不需要加 distinctselect xxx from n的表 where owner in (select owner from 1的表) 改写为 inner join 前面要加 distinct 我们的SQL 是 select count(distinct ),count(distinct) 所以 CBO 直接改写为 select count(distinct a.owner),count(distinct object_name) from a,b where a.owner=b.owner;这样就引起了 小笛卡尔积，所以就慢了 那么这个问题在 12c里面被纠正了，有兴趣自己玩个12c 试一试不管优化器多聪明，始终没有人聪明 看不懂这篇文章的人努力吧
以上是关于数据分布不均衡导致性能问题的主要内容，如果未能解决你的问题，请参考以下文章