MySQL实例多库某张表数据文件损坏导致xxx库无法访问故障恢复

Posted liyingxiao

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了MySQL实例多库某张表数据文件损坏导致xxx库无法访问故障恢复相关的知识,希望对你有一定的参考价值。

一、问题发现

  命令行进入数据库实例手动给某张表进行alter操作,发现如下报错。

mysql> use xx_xxx;
No connection. Trying to reconnect...
Connection id:    5
Current database: *** NONE ***

Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> show tables;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    3
Current database: xx_xxx

ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
ERROR 2002 (HY000): Cant connect to local MySQL server through socket /tmp/mysql.sock (111)
ERROR: 
Cant connect to the server

二、问题定位

  上述这种错误常见是MySQL实例关闭或者非正常宕机、MySQL连接超时、MySQL请求线程被kill。根据现有的业务场景,审核平台能正常连接数据库甚至连接有问题的数据库建表,MySQL服务对外正常,错误日志没有非正常输出,没有开发或者测试人员反映有问题的数据库存在使用问题。但是通过Navicat连接打开问题数据库发现报错(MySQL server has gone away),通过命令行界面进入其他数据库,执行数据库命令都正常,进入问题数据库连最基本的数据库相应变量值和状态值都无法show。

  排查暴力破解数据库尝试连接的源头,缩小问题来源(这里排查走偏了),发现问题依然存在。但是比较难理解的是通过审核平台使用问题库却能建表成功,与之前遇到的整库数据文件损坏还不一样,这里怀疑可能是某张表数据文件损坏导致了错误。

   查看日志发现实例存在异常shutdown和崩溃恢复记录,但还不能确定具体的原因,可以明确的是单个库存在问题,可以从其他途径去恢复。但是DBA存在即有价值,我们可以尽可能的先尝试以最小的代价解决问题。虽之前遇到过多次硬件故障导致的数据文件损坏,可以通过集群的其他实例和备份完成恢复并不会很大影响业务,也遇到过自己测试发现单个库文件坏得很彻底,但能通过dump出来数据文件进行恢复。

  解决问题后追溯问题,发现日志记录如下故障点(这个日志比较久远,当问题来临时可能没有那么多时间给予分析,需要快速定位初步问题并解决,解决问题的时候不一定能发现这个重要的排错依据,放上日志只是仅供参考和回溯故障原因)。

2018-12-18T12:30:29.409505Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3514141201
2018-12-18T12:30:29.409520Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3514141210
2018-12-18T12:30:29.409677Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3514141210
2018-12-18T12:30:29.409682Z 0 [Note] InnoDB: Database was not shutdown normally!
2018-12-18T12:30:29.409685Z 0 [Note] InnoDB: Starting crash recovery.
2018-12-18T12:30:30.026781Z 0 [ERROR] InnoDB: In file ‘./xx_xxxxxx/xx_xxxxxx_fans_person.ibd‘, tablespace id and flags are 2760 and 33, but in the InnoDB data dictionary they are 97
4 and 33. Have you moved InnoDB .ibd files around without using the commands DISCARD TABLESPACE and IMPORT TABLESPACE? Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb-
troubleshooting-datadict.html for how to resolve the issue.
2018-12-18T12:30:30.026819Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation.
2018-12-18T12:30:30.026823Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified.
2018-12-18T12:30:30.026827Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them.
2018-12-18T12:30:30.026831Z 0 [ERROR] InnoDB: Could not find a valid tablespace file for `xx_xxxxxx/xx_xxxxxx_fans_person`. Please refer to http://dev.mysql.com/doc/refman/5.7/en/in
nodb-troubleshooting-datadict.html for how to resolve the issue.
2018-12-18T12:30:30.026841Z 0 [Warning] InnoDB: Ignoring tablespace `xx_xxxxxx/xx_xxxxxx_fans_person` because it could not be opened.
2018-12-18T12:30:32.199013Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-12-18T12:30:32.199035Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-12-18T12:30:32.199088Z 0 [Note] InnoDB: Setting file ‘./ibtmp1‘ size to 12 MB. Physically writing the file full; Please wait ...
2018-12-18T12:30:32.286423Z 0 [Note] InnoDB: File ‘./ibtmp1‘ size is now 12 MB.

[ERROR] [FATAL] InnoDB: Tablespace id is 974 in the data dictionary but in file ./xx_xxxxxx/xx_xxxxxx_fans_person.ibd it is 2760! 2018-12-18 20:30:29 0x7f014872b700 InnoDB: Assertion failure in thread 139643487172352 in file ut0ut.cc line 916 InnoDB: We intentionally generate a memory trap. 2018-12-18T12:30:29.409505Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 3514141201 2018-12-18T12:30:29.409520Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3514141210 2018-12-18T12:30:29.409677Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 3514141210 2018-12-18T12:30:29.409682Z 0 [Note] InnoDB: Database was not shutdown normally! 2018-12-18T12:30:29.409685Z 0 [Note] InnoDB: Starting crash recovery. 2018-12-18T12:30:30.026781Z 0 [ERROR] InnoDB: In file ‘./xx_xxxxxx/xx_xxxxxx_fans_person.ibd‘, tablespace id and flags are 2760 and 33, but in the InnoDB data dictionary they are 97 4 and 33. Have you moved InnoDB .ibd files around without using the commands DISCARD TABLESPACE and IMPORT TABLESPACE? Please refer to http://dev.mysql.com/doc/refman/5.7/en/innodb- troubleshooting-datadict.html for how to resolve the issue. 2018-12-18T12:30:30.026819Z 0 [ERROR] InnoDB: Operating system error number 2 in a file operation. 2018-12-18T12:30:30.026823Z 0 [ERROR] InnoDB: The error means the system cannot find the path specified. 2018-12-18T12:30:30.026827Z 0 [ERROR] InnoDB: If you are installing InnoDB, remember that you must create directories yourself, InnoDB does not create them. 2018-12-18T12:30:30.026831Z 0 [ERROR] InnoDB: Could not find a valid tablespace file for `xx_xxxxxx/xx_xxxxxx_fans_person`. Please refer to http://dev.mysql.com/doc/refman/5.7/en/in nodb-troubleshooting-datadict.html for how to resolve the issue. 2018-12-18T12:30:30.026841Z 0 [Warning] InnoDB: Ignoring tablespace `xx_xxxxxx/xx_xxxxxx_fans_person` because it could not be opened. 2018-12-18T12:30:32.199013Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1" 2018-12-18T12:30:32.199035Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2018-12-18T12:30:32.199088Z 0 [Note] InnoDB: Setting file ‘./ibtmp1‘ size to 12 MB. Physically writing the file full; Please wait ... 2018-12-18T12:30:32.286423Z 0 [Note] InnoDB: File ‘./ibtmp1‘ size is now 12 MB.

三、问题解决

  遇到数据文件损坏导致的数据无法正常存取,通常解决的办法是通过备份进行恢复,包括对坏点进行备份恢复。尝试过才知道有思路是好的,但是实践起来不一定容易,果不其然当我想通过dump备份数据再尝试修复的时候出现了错误 MySQL server has gone away。遇到好的问题就要分享,往往问题比较宽泛不好定位的时候容易忽略正确的处理方向。通过好朋友圈的提醒,发现use库的时候输出了-A选项,查询得知可以不加载元数据信息就能进入数据库。

-A选项意义
    当我们打开数据库,即use dbname时,要预读数据库信息。由于数据库太大,即数据库中表非常多,所以如果预读数据库信息,将非常慢,所以就卡住了,如果数据库中表非常少,将不会出现问题

  幸运的是通过不预读数据可以正常查看当前数据库所有表、系统变量值和状态值,然后尝试通过对InnoDB和MyISAM表进行批量修复,不过在此应该通过select...into的方式做好数据备份,这里因为是测试环境且有相应的冗余环境,就没做备份处理再修复。通过如下命令查询所有的base表并拼接SQL语句,果然发现了无法修复的坏表,印证了MySQL错误日志的信息。

##批量修复MyISAM表
select concat(repair table ,table_name,;) from information_schema.tables where table_schema = xx_xxxx and table_type = BASE TABLE and engine = MyISAM;
 
#批量修复InnoDB表
select concat(optimize table ,table_name,;) from information_schema.tables where table_schema = xx_xxxx and table_type = BASE TABLE and engine = InnoDB;

##### optimize table xx_xxxxxx_fans_person;  

  通过上述命令发现修复结果不OK的表,并通过查看表行数确认数据已无法导出,删除相应的坏表并重新建立新表(drop table可能出现表不存在或者建表1068错误),导入最近的一次数据备份,重启MySQL实例,发现问题解决,问题库可以正常访问。

1、删除错误表xx_xxxxxx_fans_person
2、重建表
mysql>   CREATE TABLE `xx_xxxxxx_fans_person` (
    ->   `person_id` int(20) NOT NULL AUTO_INCREMENT,
    ->   `person_circle_id` int(20) NOT NULL,
    ->   `person_user_id` int(20) NOT NULL,
    ->   `person_time` datetime NOT NULL,
    ->   `type` int(4) DEFAULT 1 COMMENT 1. 组长 2. 成员,
    ->   `merchant_id` int(11) DEFAULT 0,
    ->   `leave_type` int(11) DEFAULT 0 COMMENT 请假状态 0.未请假 1.请假,
    ->   `leave_start_time` datetime DEFAULT NULL COMMENT 请假开始时间,
    ->   `leave_end_time` datetime DEFAULT NULL COMMENT 请假结束时间,
    ->   `is_invalid` int(10) DEFAULT 0 COMMENT 是否失效 0有效 1失效,
    ->   `invalid_id` int(10) DEFAULT 0 COMMENT 失效记录关联ID ,
    ->   PRIMARY KEY (`person_id`),
    ->   KEY `person_circle_id` (`person_circle_id`) USING BTREE,
    ->   KEY `person_user_id` (`person_user_id`) USING BTREE
    -> ) ENGINE=InnoDB;
ERROR 1030 (HY000): Got error 168 from storage engine
3、重启
mysql> select count(*) from xx_xxxxxx_fans_person;
ERROR 1812 (HY000): Tablespace is missing for table `xx_xxxxxx`.`xx_xxxxxx_fans_person`.

mysql> drop table xx_xxxxxx_fans_person;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TABLE `xx_xxxxxx_fans_person` (
    ->   `person_id` int(20) NOT NULL AUTO_INCREMENT,
    ->   `person_circle_id` int(20) NOT NULL,
    ->   `person_user_id` int(20) NOT NULL,
    ->   `person_time` datetime NOT NULL,
    ->   `type` int(4) DEFAULT 1 COMMENT 1. 组长 2. 成员,
    ->   `merchant_id` int(11) DEFAULT 0,
    ->   `leave_type` int(11) DEFAULT 0 COMMENT 请假状态 0.未请假 1.请假,
    ->   `leave_start_time` datetime DEFAULT NULL COMMENT 请假开始时间,
    ->   `leave_end_time` datetime DEFAULT NULL COMMENT 请假结束时间,
    ->   `is_invalid` int(10) DEFAULT 0 COMMENT 是否失效 0有效 1失效,
    ->   `invalid_id` int(10) DEFAULT 0 COMMENT 失效记录关联ID ,
    ->   PRIMARY KEY (`person_id`),
    ->   KEY `person_circle_id` (`person_circle_id`) USING BTREE,
    ->   KEY `person_user_id` (`person_user_id`) USING BTREE
    -> ) ENGINE=InnoDB;
ERROR 1813 (HY000): Tablespace `xx_xxxxxx`.`xx_xxxxxx_fans_person` exists.

四、总结

  1、数据库需要定时备份,防止硬件或者其他问题导致的数据文件损坏

  2、先分析问题,排查基本的不可能点,必需查看日志分析问题,注意查看命令报错的输出提示信息(可能帮助我们排查或者修复)

       3、可使用-A选项不加载数据库信息尝试进行表修复,提前做好备份

以上是关于MySQL实例多库某张表数据文件损坏导致xxx库无法访问故障恢复的主要内容,如果未能解决你的问题,请参考以下文章

使MySql某用户只能看到某个数据库的某张表

快速修改MySQL某张表的表结构

mysql创建某个数据库中的某张表 只读用户

MySQL数据迁移工具的设计与实现

MySQL数据迁移工具的设计与实现

如何用php判断mysql数据库里某张表是不是存在