MySQL SQL 优化参数 引发的悲剧

Posted 2021-12-16 老叶茶馆_

大家好，我是知数堂SQL 优化班老师 网名：骑龟的兔子

今天给大家看一个案例来讨论，这个案例是真实案例，因为之前踩bug

导致数据库crash 所以临时关了优化器参数

set gloabl optimizer_switch='materialization=off';

这也是可以写一篇很长的文章，但是以后再说。

上面参数是控制 semi join 的几种方式的一种方式。

但是因为更改了上面的参数，导致的结果就是本来线上运行的很好的一个SQL运行了很长时间也运行不出来，然后我分析之后发现了一个之前讲课的时候不一样的东西，所以在这里分享。

materialization 这个参数简单来说是一个对于in 

版本升级之后可以对 exists 产生作用

如下所示 有点类似于产生临时表然后可以生成临时索引

[e-mail]>[employees]>desc select   * from employees d   where  d.emp_no in ( select /*+ QB_NAME(sub1) */   t.emp_no  from dept_emp t ) ;         
+----+--------------+-------------+------------+--------+---------------------+---------------------+---------+--------------------+--------+----------+-------------+
| id | select_type  | table       | partitions | type   | possible_keys       | key                 | key_len | ref                | rows   | filtered | Extra       |
+----+--------------+-------------+------------+--------+---------------------+---------------------+---------+--------------------+--------+----------+-------------+
|  1 | SIMPLE       | d           | NULL       | ALL    | PRIMARY             | NULL                | NULL    | NULL               | 298980 |   100.00 | NULL        |
|  1 | SIMPLE       | <subquery2> | NULL       | eq_ref | <auto_distinct_key> | <auto_distinct_key> | 4       | employees.d.emp_no |      1 |   100.00 | NULL        |
|  2 | MATERIALIZED | t           | NULL       | index  | PRIMARY,emp_no      | emp_no              | 4       | NULL               | 331143 |   100.00 | Using index |
+----+--------------+-------------+------------+--------+---------------------+---------------------+---------+--------------------+--------+----------+-------------+


[e-mail]>[employees]>desc select  * from dept_emp d where   d.emp_no in (select t.emp_no  from t_group t ) ;
+----+--------------+-------------+------------+------+----------------+---------+---------+--------------------+------+----------+-------+
| id | select_type  | table       | partitions | type | possible_keys  | key     | key_len | ref                | rows | filtered | Extra |
+----+--------------+-------------+------------+------+----------------+---------+---------+--------------------+------+----------+-------+
|  1 | SIMPLE       | <subquery2> | NULL       | ALL  | NULL           | NULL    | NULL    | NULL               | NULL |   100.00 | NULL  |
|  1 | SIMPLE       | d           | NULL       | ref  | PRIMARY,emp_no | PRIMARY | 4       | <subquery2>.emp_no |    1 |   100.00 | NULL  |
|  2 | MATERIALIZED | t           | NULL       | ALL  | NULL           | NULL    | NULL    | NULL               |   10 |   100.00 | NULL  |
+----+--------------+-------------+------------+------+----------------+---------+---------+--------------------+------+----------+-------+

上面的内容在课上已经详细细说，他的优缺点和运用方法，在这里不细说

现在开始说的是最新发现的内容

有一种SQL如果是下面所展示 效果也不错

[e-mail]>[employees]>set session optimizer_switch='materialization=on';
Query OK, 0 rows affected (0.00 sec)


[e-mail]>[employees]>desc select straight_join  * from dept_emp d where   d.emp_no in (select t.emp_no  from t_group t ) ;
+----+-------------+-------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key  | key_len | ref  | rows   | filtered | Extra       |
+----+-------------+-------+------------+------+---------------+------+---------+------+--------+----------+-------------+
|  1 | PRIMARY     | d     | NULL       | ALL  | NULL          | NULL | NULL    | NULL | 331143 |   100.00 | Using where |
|  2 | SUBQUERY    | t     | NULL       | ALL  | NULL          | NULL | NULL    | NULL |     10 |   100.00 | NULL        |
+----+-------------+-------+------------+------+---------------+------+---------+------+--------+----------+-------------+


[e-mail]>[employees]>select straight_join  * from dept_emp d where   d.emp_no in (select t.emp_no  from t_group t ) ;
+--------+---------+------------+------------+
| emp_no | dept_no | from_date  | to_date    |
+--------+---------+------------+------------+
|  10004 | d004    | 1986-12-01 | 9999-01-01 |
|22744| d006    |1986-12-01| 9999-01-01 |
|  24007 | d005    | 1986-12-01 | 9999-01-01 |
|30970| d005    |1986-12-01| 9999-01-01 |
|  31112 | d002    | 1986-12-01 | 1993-12-10 |
|40983| d005    |1986-12-01| 9999-01-01 |
|  46554 | d008    | 1986-12-01 | 1992-05-27 |
|48317| d004    |1989-01-11| 9999-01-01 |
|  48317 | d008    | 1986-12-01 | 1989-01-11 |
|49667| d007    |1986-12-01| 9999-01-01 |
|  50449 | d005    | 1986-12-01 | 9999-01-01 |
+--------+---------+------------+------------+
11 rows in set (0.24 sec)

在这种情况执行由于种种原因我们把参数关了 那就会产生悲剧事件

[e-mail]>[employees]>set session optimizer_switch='materialization=off';
Query OK, 0 rows affected (0.00 sec)


[e-mail]>[employees]>desc select straight_join  * from dept_emp d where   d.emp_no in (select t.emp_no  from t_group t ) ;
+----+--------------------+-------+------------+------+---------------+------+---------+------+--------+----------+-------------+
| id | select_type        | table | partitions | type | possible_keys | key  | key_len | ref  | rows   | filtered | Extra       |
+----+--------------------+-------+------------+------+---------------+------+---------+------+--------+----------+-------------+
|1| PRIMARY            | d     | NULL       | ALL  | NULL          | NULL | NULL    | NULL | 331143 |100.00| Using where |
|  2 | DEPENDENT SUBQUERY | t     | NULL       | ALL  | NULL          | NULL | NULL    | NULL |     10 |    10.00 | Using where |
+----+--------------------+-------+------------+------+---------------+------+---------+------+--------+----------+-------------+
2 rows in set, 1 warning (0.00 sec)


[e-mail]>[employees]>select straight_join  * from dept_emp d where   d.emp_no in (select t.emp_no  from t_group t ) ;
+--------+---------+------------+------------+
| emp_no | dept_no | from_date  | to_date    |
+--------+---------+------------+------------+
|  10004 | d004    | 1986-12-01 | 9999-01-01 |
|22744| d006    |1986-12-01| 9999-01-01 |
|  24007 | d005    | 1986-12-01 | 9999-01-01 |
|30970| d005    |1986-12-01| 9999-01-01 |
|  31112 | d002    | 1986-12-01 | 1993-12-10 |
|40983| d005    |1986-12-01| 9999-01-01 |
|  46554 | d008    | 1986-12-01 | 1992-05-27 |
|48317| d004    |1989-01-11| 9999-01-01 |
|  48317 | d008    | 1986-12-01 | 1989-01-11 |
|49667| d007    |1986-12-01| 9999-01-01 |
|  50449 | d005    | 1986-12-01 | 9999-01-01 |
+--------+---------+------------+------------+
11 rows in set (2.81 sec)

运行效果如上所示性能大幅度下降

这里的重要发现是我们之前所说的materialization 这个参数竟然还可以控制  SUBQUERY

那这种情况下 怎么弄可以优化呢 

如果生成索引 性能可以恢复

[e-mail]>[employees]>create index idx_t_group_emp_no on t_group(emp_no);
Query OK, 0 rows affected (0.06 sec)
Records: 0  Duplicates: 0  Warnings: 0


 [e-mail]>[employees]>desc select straight_join  * from dept_emp d where   d.emp_no in (select t.emp_no  from t_group t ) ;
+----+--------------------+-------+------------+----------------+--------------------+--------------------+---------+------+--------+----------+-------------+
| id | select_type        | table | partitions | type           | possible_keys      | key                | key_len | ref  | rows   | filtered | Extra       |
+----+--------------------+-------+------------+----------------+--------------------+--------------------+---------+------+--------+----------+-------------+
|1| PRIMARY            | d     | NULL       | ALL            | NULL               | NULL               | NULL    | NULL | 331143 |100.00| Using where |
|  2 | DEPENDENT SUBQUERY | t     | NULL       | index_subquery | idx_t_group_emp_no | idx_t_group_emp_no | 4       | func |      1 |   100.00 | Using index |
+----+--------------------+-------+------------+----------------+--------------------+--------------------+---------+------+--------+----------+-------------+
2 rows in set, 1 warning (0.00 sec)




[e-mail]>[employees]>select straight_join  * from dept_emp d where exists (select t.emp_no  from t_group t where t.emp_no=d.emp_no ) ;
+--------+---------+------------+------------+
| emp_no | dept_no | from_date  | to_date    |
+--------+---------+------------+------------+
|  10004 | d004    | 1986-12-01 | 9999-01-01 |
|22744| d006    |1986-12-01| 9999-01-01 |
|  24007 | d005    | 1986-12-01 | 9999-01-01 |
|30970| d005    |1986-12-01| 9999-01-01 |
|  31112 | d002    | 1986-12-01 | 1993-12-10 |
|40983| d005    |1986-12-01| 9999-01-01 |
|  46554 | d008    | 1986-12-01 | 1992-05-27 |
|48317| d004    |1989-01-11| 9999-01-01 |
|  48317 | d008    | 1986-12-01 | 1989-01-11 |
|49667| d007    |1986-12-01| 9999-01-01 |
|  50449 | d005    | 1986-12-01 | 9999-01-01 |
+--------+---------+------------+------------+
11 rows in set (0.84 sec)

index_subquery

This join type is similar to unique_subquery. It replaces IN subqueries, but it works for nonunique indexes in subqueries of the following form:

value IN (SELECT key_column FROM single_table WHERE some_expr)

这就本文想说的所有内容

materialization  这个参数实际运维过程中，一些稀奇古怪的数据不对的，甚至宕机的现象 也有可能跟这个参数有关，但是知悉分析之后可以发下还是SQL 写的有问题，可以通过修改SQL 形式绕开，大家多留意一下

P.S，松华老师遇到的Bug截图见下：

官方的回复，建议升级到最新版本，目前是8.0.27。此外，强烈建议至少升级到8.0.20之后，在此之前的版本某公有云上遇到非常多的问题。

我是知数堂SQL 优化班老师~ ^^

最新一期SQL优化课，在12月份开始。

如有关于SQL优化方面疑问和一起交流的请加 并且 @兔子@知数堂SQL优化

高性能MySQL,SQL优化群 有叶金荣，吴炳锡 两位大神坐镇 ：579036588

欢迎加入 知数堂大家庭。

我的微信公众号：SQL开发与优化(sqlturning)

《实战MGR》视频课程

戳此小程序即可直达B站或腾讯课堂

或复制链接在浏览器中打开

• https://ke.qq.com/course/3677969

• https://space.bilibili.com/1363850082

---

文章推荐：

• 面向金融级应用的GreatSQL正式开源
  

• Changes in GreatSQL 8.0.25 (2021-8-26)

• GreatSQL重磅特性，InnoDB并行查询优化测试

• MySQL DDL简析（1）：inplace DDL 主要stage介绍
  

• Percona XtraBackup 8.0.26实战大全

• 一条sql语句慢在哪之抓包分析

• 技术分享 | Update更新慢、死锁等问题的排查思路分享
  

• GreatSQL重磅特性，InnoDB并行并行查询优化测试
  

• 在Linux下源码编译安装GreatSQL/MySQL

---

点击文末“阅读原文”直达老叶专栏