MySQL实例多库某张表数据文件损坏导致xxx库无法访问故障恢复

Posted 2022-03-10 liyingxiao

一、问题发现

　　命令行进入数据库实例手动给某张表进行alter操作，发现如下报错。

mysql> use xx_xxx;
No connection. Trying to reconnect...
Connection id:    5
Current database: *** NONE ***

Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    3
Current database: xx_xxx

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Can‘t connect to local MySQL server through socket ‘/tmp/mysql.sock‘ (111)
ERROR: 
Can‘t connect to the server

二、问题定位

　　上述这种错误常见是MySQL实例关闭或者非正常宕机、MySQL连接超时、MySQL请求线程被kill。根据现有的业务场景，审核平台能正常连接数据库甚至连接有问题的数据库建表，MySQL服务对外正常，错误日志没有非正常输出，没有开发或者测试人员反映有问题的数据库存在使用问题。但是通过Navicat连接打开问题数据库发现报错（MySQL server has gone away），通过命令行界面进入其他数据库，执行数据库命令都正常，进入问题数据库连最基本的数据库相应变量值和状态值都无法show。

　　排查暴力破解数据库尝试连接的源头，缩小问题来源（这里排查走偏了），发现问题依然存在。但是比较难理解的是通过审核平台使用问题库却能建表成功，与之前遇到的整库数据文件损坏还不一样，这里怀疑可能是某张表数据文件损坏导致了错误。

 　　查看日志发现实例存在异常shutdown和崩溃恢复记录，但还不能确定具体的原因，可以明确的是单个库存在问题，可以从其他途径去恢复。但是DBA存在即有价值，我们可以尽可能的先尝试以最小的代价解决问题。虽之前遇到过多次硬件故障导致的数据文件损坏，可以通过集群的其他实例和备份完成恢复并不会很大影响业务，也遇到过自己测试发现单个库文件坏得很彻底，但能通过dump出来数据文件进行恢复。

　　解决问题后追溯问题，发现日志记录如下故障点（这个日志比较久远，当问题来临时可能没有那么多时间给予分析，需要快速定位初步问题并解决，解决问题的时候不一定能发现这个重要的排错依据，放上日志只是仅供参考和回溯故障原因）。

2018-12-18T12:30:29.409505Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3514141201
2018-12-18T12:30:29.409520Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3514141210
2018-12-18T12:30:29.409677Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3514141210
2018-12-18T12:30:29.409682Z 0 [Note] InnoDB: Database was not shutdown normally!
2018-12-18T12:30:29.409685Z 0 [Note] InnoDB: Starting crash recovery.
2018-12-18T12:30:30.026781Z 0 [ERROR] InnoDB: In file ‘./xx_xxxxxx/xx_xxxxxx_fans_person.ibd‘, tablespace id and flags are 2760 and 33, but in the InnoDB data dictionary they are 97
4 and 33. Have you moved InnoDB .ibd files around without using the commands DISCARD TABLESPACE and IMPORT TABLESPACE? Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-
troubleshooting-datadict.html for how to resolve the issue.
2018-12-18T12:30:30.026819Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2018-12-18T12:30:30.026823Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2018-12-18T12:30:30.026827Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
2018-12-18T12:30:30.026831Z 0 [ERROR] InnoDB: Could not find a valid tablespace file for `xx_xxxxxx/xx_xxxxxx_fans_person`. Please refer to http://dev.mysql.com/doc/refman/5.7/en/in
nodb-troubleshooting-datadict.html for how to resolve the issue.
2018-12-18T12:30:30.026841Z 0 [Warning] InnoDB: Ignoring tablespace `xx_xxxxxx/xx_xxxxxx_fans_person` because it could not be opened.
2018-12-18T12:30:32.199013Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-12-18T12:30:32.199035Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-12-18T12:30:32.199088Z 0 [Note] InnoDB: Setting file ‘./ibtmp1‘ size to 12 MB. Physically writing the file full; Please wait ...
2018-12-18T12:30:32.286423Z 0 [Note] InnoDB: File ‘./ibtmp1‘ size is now 12 MB.

[ERROR] [FATAL] InnoDB: Tablespace id is 974 in the data dictionary but in file ./xx_xxxxxx/xx_xxxxxx_fans_person.ibd it is 2760!
2018-12-18 20:30:29 0x7f014872b700  InnoDB: Assertion failure in thread 139643487172352 in file ut0ut.cc line 916
InnoDB: We intentionally generate a memory trap.

2018-12-18T12:30:29.409505Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3514141201
2018-12-18T12:30:29.409520Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3514141210
2018-12-18T12:30:29.409677Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3514141210
2018-12-18T12:30:29.409682Z 0 [Note] InnoDB: Database was not shutdown normally!
2018-12-18T12:30:29.409685Z 0 [Note] InnoDB: Starting crash recovery.
2018-12-18T12:30:30.026781Z 0 [ERROR] InnoDB: In file ‘./xx_xxxxxx/xx_xxxxxx_fans_person.ibd‘, tablespace id and flags are 2760 and 33, but in the InnoDB data dictionary they are 97
4 and 33. Have you moved InnoDB .ibd files around without using the commands DISCARD TABLESPACE and IMPORT TABLESPACE? Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-
troubleshooting-datadict.html for how to resolve the issue.
2018-12-18T12:30:30.026819Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2018-12-18T12:30:30.026823Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2018-12-18T12:30:30.026827Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
2018-12-18T12:30:30.026831Z 0 [ERROR] InnoDB: Could not find a valid tablespace file for `xx_xxxxxx/xx_xxxxxx_fans_person`. Please refer to http://dev.mysql.com/doc/refman/5.7/en/in
nodb-troubleshooting-datadict.html for how to resolve the issue.
2018-12-18T12:30:30.026841Z 0 [Warning] InnoDB: Ignoring tablespace `xx_xxxxxx/xx_xxxxxx_fans_person` because it could not be opened.
2018-12-18T12:30:32.199013Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-12-18T12:30:32.199035Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-12-18T12:30:32.199088Z 0 [Note] InnoDB: Setting file ‘./ibtmp1‘ size to 12 MB. Physically writing the file full; Please wait ...
2018-12-18T12:30:32.286423Z 0 [Note] InnoDB: File ‘./ibtmp1‘ size is now 12 MB.

三、问题解决

　　遇到数据文件损坏导致的数据无法正常存取，通常解决的办法是通过备份进行恢复，包括对坏点进行备份恢复。尝试过才知道有思路是好的，但是实践起来不一定容易，果不其然当我想通过dump备份数据再尝试修复的时候出现了错误 MySQL server has gone away。遇到好的问题就要分享，往往问题比较宽泛不好定位的时候容易忽略正确的处理方向。通过好朋友圈的提醒，发现use库的时候输出了-A选项，查询得知可以不加载元数据信息就能进入数据库。

-A选项意义
    当我们打开数据库，即use dbname时，要预读数据库信息。由于数据库太大，即数据库中表非常多，所以如果预读数据库信息，将非常慢，所以就卡住了，如果数据库中表非常少，将不会出现问题

　　幸运的是通过不预读数据可以正常查看当前数据库所有表、系统变量值和状态值，然后尝试通过对InnoDB和MyISAM表进行批量修复，不过在此应该通过select...into的方式做好数据备份，这里因为是测试环境且有相应的冗余环境，就没做备份处理再修复。通过如下命令查询所有的base表并拼接SQL语句，果然发现了无法修复的坏表，印证了MySQL错误日志的信息。

##批量修复MyISAM表
select concat(‘repair table ‘,table_name,‘;‘) from information_schema.tables where table_schema = ‘xx_xxxx‘ and table_type = ‘BASE TABLE‘ and engine = ‘MyISAM‘;
 
#批量修复InnoDB表
select concat(‘optimize table ‘,table_name,‘;‘) from information_schema.tables where table_schema = ‘xx_xxxx‘ and table_type = ‘BASE TABLE‘ and engine = ‘InnoDB‘;

##### optimize table xx_xxxxxx_fans_person;  

　　通过上述命令发现修复结果不OK的表，并通过查看表行数确认数据已无法导出，删除相应的坏表并重新建立新表（drop table可能出现表不存在或者建表1068错误），导入最近的一次数据备份，重启MySQL实例，发现问题解决，问题库可以正常访问。

1、删除错误表xx_xxxxxx_fans_person
2、重建表
mysql>   CREATE TABLE `xx_xxxxxx_fans_person` (
    ->   `person_id` int(20) NOT NULL AUTO_INCREMENT,
    ->   `person_circle_id` int(20) NOT NULL,
    ->   `person_user_id` int(20) NOT NULL,
    ->   `person_time` datetime NOT NULL,
    ->   `type` int(4) DEFAULT ‘1‘ COMMENT ‘1. 组长 2. 成员‘,
    ->   `merchant_id` int(11) DEFAULT ‘0‘,
    ->   `leave_type` int(11) DEFAULT ‘0‘ COMMENT ‘请假状态 0.未请假 1.请假‘,
    ->   `leave_start_time` datetime DEFAULT NULL COMMENT ‘请假开始时间‘,
    ->   `leave_end_time` datetime DEFAULT NULL COMMENT ‘请假结束时间‘,
    ->   `is_invalid` int(10) DEFAULT ‘0‘ COMMENT ‘是否失效 0有效 1失效‘,
    ->   `invalid_id` int(10) DEFAULT ‘0‘ COMMENT ‘失效记录关联ID ‘,
    ->   PRIMARY KEY (`person_id`),
    ->   KEY `person_circle_id` (`person_circle_id`) USING BTREE,
    ->   KEY `person_user_id` (`person_user_id`) USING BTREE
    -> ) ENGINE=InnoDB;
ERROR 1030 (HY000): Got error 168 from storage engine
3、重启
mysql> select count(*) from xx_xxxxxx_fans_person;
ERROR 1812 (HY000): Tablespace is missing for table `xx_xxxxxx`.`xx_xxxxxx_fans_person`.

mysql> drop table xx_xxxxxx_fans_person;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TABLE `xx_xxxxxx_fans_person` (
    ->   `person_id` int(20) NOT NULL AUTO_INCREMENT,
    ->   `person_circle_id` int(20) NOT NULL,
    ->   `person_user_id` int(20) NOT NULL,
    ->   `person_time` datetime NOT NULL,
    ->   `type` int(4) DEFAULT ‘1‘ COMMENT ‘1. 组长 2. 成员‘,
    ->   `merchant_id` int(11) DEFAULT ‘0‘,
    ->   `leave_type` int(11) DEFAULT ‘0‘ COMMENT ‘请假状态 0.未请假 1.请假‘,
    ->   `leave_start_time` datetime DEFAULT NULL COMMENT ‘请假开始时间‘,
    ->   `leave_end_time` datetime DEFAULT NULL COMMENT ‘请假结束时间‘,
    ->   `is_invalid` int(10) DEFAULT ‘0‘ COMMENT ‘是否失效 0有效 1失效‘,
    ->   `invalid_id` int(10) DEFAULT ‘0‘ COMMENT ‘失效记录关联ID ‘,
    ->   PRIMARY KEY (`person_id`),
    ->   KEY `person_circle_id` (`person_circle_id`) USING BTREE,
    ->   KEY `person_user_id` (`person_user_id`) USING BTREE
    -> ) ENGINE=InnoDB;
ERROR 1813 (HY000): Tablespace ‘`xx_xxxxxx`.`xx_xxxxxx_fans_person`‘ exists.

四、总结

　　1、数据库需要定时备份，防止硬件或者其他问题导致的数据文件损坏

　　2、先分析问题，排查基本的不可能点，必需查看日志分析问题，注意查看命令报错的输出提示信息（可能帮助我们排查或者修复）

       3、可使用-A选项不加载数据库信息尝试进行表修复，提前做好备份