记录一则mysql主从延迟故障log
Posted nadian
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了记录一则mysql主从延迟故障log相关的知识，希望对你有一定的参考价值。
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| newauth4a |
| performance_schema |
| pre4a |
| sys |
+--------------------+
6 rows in set (0.01 sec)
mysql> use information_schema
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables like ‘%trx%‘;
+--------------------------------------+
| Tables_in_information_schema (%trx%) |
+--------------------------------------+
| INNODB_TRX |
+--------------------------------------+
1 row in set (0.00 sec)
mysql> select * from information_schema.INNODB_TRX;
+--------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| trx_id | trx_state | trx_started | trx_requested_lock_id | trx_wait_started | trx_weight | trx_mysql_thread_id | trx_query | trx_operation_state | trx_tables_in_use | trx_tables_locked | trx_lock_structs | trx_lock_memory_bytes | trx_rows_locked | trx_rows_modified | trx_concurrency_tickets | trx_isolation_level | trx_unique_checks | trx_foreign_key_checks | trx_last_foreign_key_error | trx_adaptive_hash_latched | trx_adaptive_hash_timeout | trx_is_read_only | trx_autocommit_non_locking |
+--------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
| 685760 | RUNNING | 2020-06-17 01:14:51 | NULL | NULL | 6 | 180 | NULL | NULL | 0 | 2 | 4 | 1136 | 2 | 2 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 0 | 0 | 0 |
| 685759 | RUNNING | 2020-06-17 01:14:51 | NULL | NULL | 6 | 182 | NULL | NULL | 0 | 2 | 4 | 1136 | 2 | 2 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 0 | 0 | 0 |
| 685758 | RUNNING | 2020-06-17 01:14:51 | NULL | NULL | 6 | 181 | NULL | NULL | 0 | 2 | 4 | 1136 | 2 | 2 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 0 | 0 | 0 |
| 685464 | RUNNING | 2020-06-17 00:47:50 | NULL | NULL | 371995 | 179 | NULL | NULL | 0 | 2 | 5464 | 516304 | 366531 | 366531 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 0 | 0 | 0 |
| 683807 | RUNNING | 2020-06-16 14:12:48 | NULL | NULL | 763221 | 178 | NULL | NULL | 0 | 2 | 23812 | 2121936 | 739409 | 739409 | 0 | READ COMMITTED | 1 | 1 | NULL | 0 | 0 | 0 | 0 |
+--------+-----------+---------------------+-----------------------+------------------+------------+---------------------+-----------+---------------------+-------------------+-------------------+------------------+-----------------------+-----------------+-------------------+-------------------------+---------------------+-------------------+------------------------+----------------------------+---------------------------+---------------------------+------------------+----------------------------+
5 rows in set (0.01 sec)
in set (0.01 sec)
mysql> desc information_schema.INNODB_TRX
-> ;
+----------------------------+---------------------+------+-----+---------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+----------------------------+---------------------+------+-----+---------------------+-------+
| trx_id | varchar(18) | NO | | | |
| trx_state | varchar(13) | NO | | | |
| trx_started | datetime | NO | | 0000-00-00 00:00:00 | |
| trx_requested_lock_id | varchar(81) | YES | | NULL | |
| trx_wait_started | datetime | YES | | NULL | |
| trx_weight | bigint(21) unsigned | NO | | 0 | |
| trx_mysql_thread_id | bigint(21) unsigned | NO | | 0 | |
| trx_query | varchar(1024) | YES | | NULL | |
| trx_operation_state | varchar(64) | YES | | NULL | |
| trx_tables_in_use | bigint(21) unsigned | NO | | 0 | |
| trx_tables_locked | bigint(21) unsigned | NO | | 0 | |
| trx_lock_structs | bigint(21) unsigned | NO | | 0 | |
| trx_lock_memory_bytes | bigint(21) unsigned | NO | | 0 | |
| trx_rows_locked | bigint(21) unsigned | NO | | 0 | |
| trx_rows_modified | bigint(21) unsigned | NO | | 0 | |
| trx_concurrency_tickets | bigint(21) unsigned | NO | | 0 | |
| trx_isolation_level | varchar(16) | NO | | | |
| trx_unique_checks | int(1) | NO | | 0 | |
| trx_foreign_key_checks | int(1) | NO | | 0 | |
| trx_last_foreign_key_error | varchar(256) | YES | | NULL | |
| trx_adaptive_hash_latched | int(1) | NO | | 0 | |
| trx_adaptive_hash_timeout | bigint(21) unsigned | NO | | 0 | |
| trx_is_read_only | int(1) | NO | | 0 | |
| trx_autocommit_non_locking | int(1) | NO | | 0 | |
+----------------------------+---------------------+------+-----+---------------------+-------+
24 rows in set (0.00 sec)
mysql> select trx_id,trx_statetrx_started,trx_query from information_schema.INNODB_TRX;
ERROR 1054 (42S22): Unknown column ‘trx_statetrx_started‘ in ‘field list‘
好几天前的start时间
mysql> select trx_id,trx_started,trx_wait_started,trx_mysql_thread_id,trx_query,trx_isolation_level from information_schema.INNODB_TRX;
+--------+---------------------+------------------+---------------------+-----------+---------------------+
| trx_id | trx_started | trx_wait_started | trx_mysql_thread_id | trx_query | trx_isolation_level |
+--------+---------------------+------------------+---------------------+-----------+---------------------+
| 685760 | 2020-06-17 01:14:51 | NULL | 180 | NULL | READ COMMITTED |
| 685759 | 2020-06-17 01:14:51 | NULL | 182 | NULL | READ COMMITTED |
| 685758 | 2020-06-17 01:14:51 | NULL | 181 | NULL | READ COMMITTED |
| 685464 | 2020-06-17 00:47:50 | NULL | 179 | NULL | READ COMMITTED |
| 683807 | 2020-06-16 14:12:48 | NULL | 178 | NULL | READ COMMITTED |
+--------+---------------------+------------------+---------------------+-----------+---------------------+
5 rows in set (0.00 sec)
mysql> select * from information_schema.INNODB_TRXG;
*************************** 1. row ***************************
trx_id: 685760
trx_state: RUNNING
trx_started: 2020-06-17 01:14:51
trx_requested_lock_id: NULL
trx_wait_started: NULL
trx_weight: 6
trx_mysql_thread_id: 180
trx_query: NULL
trx_operation_state: NULL
trx_tables_in_use: 0
trx_tables_locked: 2
trx_lock_structs: 4
trx_lock_memory_bytes: 1136
trx_rows_locked: 2
trx_rows_modified: 2
trx_concurrency_tickets: 0
trx_isolation_level: READ COMMITTED
trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
trx_adaptive_hash_latched: 0
trx_adaptive_hash_timeout: 0
trx_is_read_only: 0
trx_autocommit_non_locking: 0
*************************** 2. row ***************************
trx_id: 685759
trx_state: RUNNING
trx_started: 2020-06-17 01:14:51
trx_requested_lock_id: NULL
trx_wait_started: NULL
trx_weight: 6
trx_mysql_thread_id: 182
trx_query: NULL
trx_operation_state: NULL
trx_tables_in_use: 0
trx_tables_locked: 2
trx_lock_structs: 4
trx_lock_memory_bytes: 1136
trx_rows_locked: 2
trx_rows_modified: 2
trx_concurrency_tickets: 0
trx_isolation_level: READ COMMITTED
trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
trx_adaptive_hash_latched: 0
trx_adaptive_hash_timeout: 0
trx_is_read_only: 0
trx_autocommit_non_locking: 0
*************************** 3. row ***************************
trx_id: 685758
trx_state: RUNNING
trx_started: 2020-06-17 01:14:51
trx_requested_lock_id: NULL
trx_wait_started: NULL
trx_weight: 6
trx_mysql_thread_id: 181
trx_query: NULL
trx_operation_state: NULL
trx_tables_in_use: 0
trx_tables_locked: 2
trx_lock_structs: 4
trx_lock_memory_bytes: 1136
trx_rows_locked: 2
trx_rows_modified: 2
trx_concurrency_tickets: 0
trx_isolation_level: READ COMMITTED
trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
trx_adaptive_hash_latched: 0
trx_adaptive_hash_timeout: 0
trx_is_read_only: 0
trx_autocommit_non_locking: 0
*************************** 4. row ***************************
trx_id: 685464
trx_state: RUNNING
trx_started: 2020-06-17 00:47:50
trx_requested_lock_id: NULL
trx_wait_started: NULL
trx_weight: 371995
trx_mysql_thread_id: 179
trx_query: NULL
trx_operation_state: NULL
trx_tables_in_use: 0
trx_tables_locked: 2
trx_lock_structs: 5464
trx_lock_memory_bytes: 516304
trx_rows_locked: 366531
trx_rows_modified: 366531
trx_concurrency_tickets: 0
trx_isolation_level: READ COMMITTED
trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
trx_adaptive_hash_latched: 0
trx_adaptive_hash_timeout: 0
trx_is_read_only: 0
trx_autocommit_non_locking: 0
*************************** 5. row ***************************
trx_id: 683807
trx_state: RUNNING
trx_started: 2020-06-16 14:12:48
trx_requested_lock_id: NULL
trx_wait_started: NULL
trx_weight: 763221
trx_mysql_thread_id: 178
trx_query: NULL
trx_operation_state: NULL
trx_tables_in_use: 0
trx_tables_locked: 2
trx_lock_structs: 23812
trx_lock_memory_bytes: 2121936
trx_rows_locked: 739409
trx_rows_modified: 739409
trx_concurrency_tickets: 0
trx_isolation_level: READ COMMITTED
trx_unique_checks: 1
trx_foreign_key_checks: 1
trx_last_foreign_key_error: NULL
trx_adaptive_hash_latched: 0
trx_adaptive_hash_timeout: 0
trx_is_read_only: 0
trx_autocommit_non_locking: 0
5 rows in set (0.01 sec)
ERROR:
No query specified
mysql> show processlist;
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+-----------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+-----------------------------+
| 11 | event_scheduler | localhost | NULL | Daemon | 2136129 | Waiting on empty queue | NULL |
| 177 | system user | | NULL | Connect | 1072626 | Waiting for dependent transaction to commit | NULL |
| 178 | system user | | NULL | Connect | 1210601 | Waiting for commit lock | NULL |
| 179 | system user | | NULL | Connect | 1210601 | Waiting for preceding transaction to commit | NULL |
| 180 | system user | | NULL | Connect | 1210566 | Waiting for preceding transaction to commit | NULL |
| 181 | system user | | NULL | Connect | 1210561 | Waiting for preceding transaction to commit | NULL |
| 182 | system user | | NULL | Connect | 1210561 | Waiting for preceding transaction to commit | NULL |
| 183 | system user | | NULL | Connect | 1210673 | Waiting for an event from Coordinator | NULL |
| 184 | system user | | NULL | Connect | 1210818 | Waiting for an event from Coordinator | NULL |
| 185 | system user | | NULL | Connect | 1210886 | Waiting for an event from Coordinator | NULL |
| 186 | system user | | NULL | Connect | 1210886 | Waiting for an event from Coordinator | NULL |
| 187 | system user | | NULL | Connect | 1210886 | Waiting for an event from Coordinator | NULL |
| 188 | system user | | NULL | Connect | 1210885 | Waiting for an event from Coordinator | NULL |
| 189 | system user | | NULL | Connect | 1210886 | Waiting for an event from Coordinator | NULL |
| 190 | system user | | NULL | Connect | 1210886 | Waiting for an event from Coordinator | NULL |
| 191 | system user | | NULL | Connect | 1210886 | Waiting for an event from Coordinator | NULL |
| 192 | system user | | NULL | Connect | 1210886 | Waiting for an event from Coordinator | NULL |
| 193 | system user | | NULL | Connect | 1210886 | Waiting for an event from Coordinator | NULL |
| 397639 | mydata_bk | 127.0.0.1:33090 | NULL | Query | 1068779 | Waiting for commit lock | FLUSH TABLES WITH READ LOCK |
| 429698 | mydata_bk | 127.0.0.1:47022 | NULL | Query | 982916 | Waiting for commit lock | FLUSH TABLES WITH READ LOCK |
| 461760 | mydata_bk | 127.0.0.1:32986 | NULL | Query | 896518 | Waiting for commit lock | FLUSH TABLES WITH READ LOCK |
| 493823 | mydata_bk | 127.0.0.1:46712 | NULL | Query | 810119 | Waiting for commit lock | FLUSH TABLES WITH READ LOCK |
| 778127 | system user | | NULL | Connect | 44084 | Waiting for master to send event | NULL |
| 793385 | root | localhost | NULL | Query | 1245 | Killing slave | stop slave |
| 794126 | root | localhost | information_schema | Query | 0 | starting | show processlist |
| 794516 | mydata_repl | 192.168.220.116:43748 | NULL | Binlog Dump | 30 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 794522 | mydata_ha | 192.168.220.114:48808 | NULL | Sleep | 3 | | NULL |
| 794523 | mydata_ha | 192.168.220.114:48810 | NULL | Sleep | 3 | | NULL |
| 794524 | mydata_ha | 192.168.220.114:48812 | NULL | Sleep | 3 | | NULL |
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+-----------------------------+
29 rows in set (0.00 sec)
mysql> kill 178
-> ;
Query OK, 0 rows affected (0.01 sec)
mysql> select trx_id,trx_started,trx_wait_started,trx_mysql_thread_id,trx_query,trx_isolation_level from information_schema.INNODB_TRX;
+--------+---------------------+------------------+---------------------+-----------+---------------------+
| trx_id | trx_started | trx_wait_started | trx_mysql_thread_id | trx_query | trx_isolation_level |
+--------+---------------------+------------------+---------------------+-----------+---------------------+
| 685760 | 2020-06-17 01:14:51 | NULL | 180 | NULL | READ COMMITTED |
| 685759 | 2020-06-17 01:14:51 | NULL | 182 | NULL | READ COMMITTED |
| 685758 | 2020-06-17 01:14:51 | NULL | 181 | NULL | READ COMMITTED |
| 685464 | 2020-06-17 00:47:50 | NULL | 179 | NULL | READ COMMITTED |
| 683807 | 2020-06-16 14:12:48 | NULL | 178 | NULL | READ COMMITTED |
+--------+---------------------+------------------+---------------------+-----------+---------------------+
5 rows in set (0.00 sec)
mysql> show processlist;
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+------------------+
| 11 | event_scheduler | localhost | NULL | Daemon | 2136527 | Waiting on empty queue | NULL |
| 778127 | system user | | NULL | Connect | 44482 | Waiting for master to send event | NULL |
| 794126 | root | localhost | information_schema | Query | 0 | starting | show processlist |
| 794605 | mydata_repl | 192.168.220.116:44510 | NULL | Binlog Dump | 183 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 794669 | mydata_ha | 192.168.220.114:52584 | NULL | Sleep | 5 | | NULL |
| 794670 | mydata_ha | 192.168.220.114:52586 | NULL | Sleep | 5 | | NULL |
| 794671 | mydata_ha | 192.168.220.114:52588 | NULL | Sleep | 5 | | NULL |
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+------------------+
7 rows in set (0.00 sec)
mysql> show processlist;
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+------------------+
| 11 | event_scheduler | localhost | NULL | Daemon | 2136618 | Waiting on empty queue | NULL |
| 778127 | system user | | NULL | Connect | 44573 | Waiting for master to send event | NULL |
| 794126 | root | localhost | information_schema | Query | 0 | starting | show processlist |
| 794691 | root | localhost | NULL | Sleep | 42 | | NULL |
| 794701 | mydata_repl | 192.168.220.116:45270 | NULL | Binlog Dump | 29 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 794707 | mydata_ha | 192.168.220.114:56650 | NULL | Sleep | 6 | | NULL |
| 794708 | mydata_ha | 192.168.220.114:56652 | NULL | Sleep | 6 | | NULL |
| 794709 | mydata_ha | 192.168.220.114:56654 | NULL | Sleep | 6 | | NULL |
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+------------------+
8 rows in set (0.00 sec)
mysql> show slave statusG;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.220.114
Master_User: mydata_repl
Master_Port: 10000
Connect_Retry: 60
Master_Log_File: mysql-bin.000030
Read_Master_Log_Pos: 524936381
Relay_Log_File: mysql-relay.000058
Relay_Log_Pos: 208143277
Relay_Master_Log_File: mysql-bin.000022
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1317
Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction ‘c64a3775-a630-11ea-bb58-94f128adbb08:273713‘ at master log mysql-bin.000022, end_log_pos 544224788. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
Skip_Counter: 0
Exec_Master_Log_Pos: 208143064
Relay_Log_Space: 9461331822
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1317
Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction ‘c64a3775-a630-11ea-bb58-94f128adbb08:273713‘ at master log mysql-bin.000022, end_log_pos 544224788. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
Replicate_Ignore_Server_Ids:
Master_Server_Id: 200604011
Master_UUID: c64a3775-a630-11ea-bb58-94f128adbb08
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 200629 11:18:27
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: c64a3775-a630-11ea-bb58-94f128adbb08:1-3000912
Executed_Gtid_Set: 9f018806-a625-11ea-85e2-94f128adc870:1-2,
c64a3775-a630-11ea-bb58-94f128adbb08:1-273712,
d7abc985-a626-11ea-8aaa-94f128adb6e0:1-36
Auto_Position: 1
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
ERROR:
No query specified
重启sql线程
mysql> start slave sql_thread;
Query OK, 0 rows affected (0.06 sec)
mysql> show slave statusG;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.220.114
Master_User: mydata_repl
Master_Port: 10000
Connect_Retry: 60
Master_Log_File: mysql-bin.000030
Read_Master_Log_Pos: 525101140
Relay_Log_File: mysql-relay.000058
Relay_Log_Pos: 208143277
Relay_Master_Log_File: mysql-bin.000022
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 208143064
Relay_Log_Space: 9461496581
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 1210644
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 200604011
Master_UUID: c64a3775-a630-11ea-bb58-94f128adbb08
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Waiting for Slave Workers to free pending events
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: c64a3775-a630-11ea-bb58-94f128adbb08:1-3001018
Executed_Gtid_Set: 9f018806-a625-11ea-85e2-94f128adc870:1-2,
c64a3775-a630-11ea-bb58-94f128adbb08:1-273712,
d7abc985-a626-11ea-8aaa-94f128adb6e0:1-36
Auto_Position: 1
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
ERROR:
No query specified
mysql> select trx_id,trx_started,trx_wait_started,trx_mysql_thread_id,trx_query,trx_isolation_level from information_schema.INNODB_TRX;
+--------+---------------------+------------------+---------------------+--------------------------------+---------------------+
| trx_id | trx_started | trx_wait_started | trx_mysql_thread_id | trx_query | trx_isolation_level |
+--------+---------------------+------------------+---------------------+--------------------------------+---------------------+
| 736352 | 2020-06-29 11:21:54 | NULL | 794751 | delete from S_MASTER | READ COMMITTED |
+--------+---------------------+------------------+---------------------+--------------------------------+---------------------+
1 row in set (0.00 sec)
mysql> show processlist;
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+--------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+--------------------------------+
| 11 | event_scheduler | localhost | NULL | Daemon | 2136775 | Waiting on empty queue | NULL |
| 778127 | system user | | NULL | Connect | 44730 | Waiting for master to send event | NULL |
| 794126 | root | localhost | information_schema | Query | 0 | starting | show processlist |
| 794691 | root | localhost | NULL | Sleep | 35 | | NULL |
| 794701 | mydata_repl | 192.168.220.116:45270 | NULL | Binlog Dump | 186 | Master has sent all binlog to slave; waiting for more updates | NULL |
| 794750 | system user | | NULL | Connect | 0 | Waiting for Slave Workers to free pending events | NULL |
| 794751 | system user | | NULL | Connect | 1211247 | Executing event | delete from S_MASTER |
| 794752 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794753 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794754 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794755 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794756 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794757 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794758 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794759 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794760 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794761 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794762 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794763 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794764 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794765 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794766 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL |
| 794781 | mydata_ha | 192.168.220.114:58140 | NULL | Sleep | 1 | | NULL |
| 794782 | mydata_ha | 192.168.220.114:58142 | NULL | Sleep | 1 | | NULL |
| 794783 | mydata_ha | 192.168.220.114:58144 | NULL | Sleep | 1 | | NULL |
+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+--------------------------------+
25 rows in set (0.00 sec)
mysql> show slave statusG;*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 133.96.30.141 Master_User: mydata_repl Master_Port: 10000 Connect_Retry: 60 Master_Log_File: mysql-bin.000030 Read_Master_Log_Pos: 524936381 Relay_Log_File: mysql-relay.000058 Relay_Log_Pos: 208143277 Relay_Master_Log_File: mysql-bin.000022 Slave_IO_Running: Yes Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 1317 Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction ‘c64a3775-a630-11ea-bb58-94f128adbb08:273713‘ at master log mysql-bin.000022, end_log_pos 544224788. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any. Skip_Counter: 0 Exec_Master_Log_Pos: 208143064 Relay_Log_Space: 9461331822 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULLMaster_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 1317 Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction ‘c64a3775-a630-11ea-bb58-94f128adbb08:273713‘ at master log mysql-bin.000022, end_log_pos 544224788. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any. Replicate_Ignore_Server_Ids: Master_Server_Id: 200604011 Master_UUID: c64a3775-a630-11ea-bb58-94f128adbb08 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: 200629 11:18:27 Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: c64a3775-a630-11ea-bb58-94f128adbb08:1-3000912 Executed_Gtid_Set: 9f018806-a625-11ea-85e2-94f128adc870:1-2,c64a3775-a630-11ea-bb58-94f128adbb08:1-273712,d7abc985-a626-11ea-8aaa-94f128adb6e0:1-36 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec)
ERROR: No query specified
mysql> start slave sql_thread;
Query OK, 0 rows affected (0.06 sec)
mysql> show slave statusG;*************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 133.96.30.141 Master_User: mydata_repl Master_Port: 10000 Connect_Retry: 60 Master_Log_File: mysql-bin.000030 Read_Master_Log_Pos: 525101140 Relay_Log_File: mysql-relay.000058 Relay_Log_Pos: 208143277 Relay_Master_Log_File: mysql-bin.000022 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 208143064 Relay_Log_Space: 9461496581 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 1210644Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 200604011 Master_UUID: c64a3775-a630-11ea-bb58-94f128adbb08 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Waiting for Slave Workers to free pending events Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: c64a3775-a630-11ea-bb58-94f128adbb08:1-3001018 Executed_Gtid_Set: 9f018806-a625-11ea-85e2-94f128adc870:1-2,c64a3775-a630-11ea-bb58-94f128adbb08:1-273712,d7abc985-a626-11ea-8aaa-94f128adb6e0:1-36 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec)
ERROR: No query specified
mysql> select trx_id,trx_started,trx_wait_started,trx_mysql_thread_id,trx_query,trx_isolation_level from information_schema.INNODB_TRX;+--------+---------------------+------------------+---------------------+--------------------------------+---------------------+| trx_id | trx_started | trx_wait_started | trx_mysql_thread_id | trx_query | trx_isolation_level |+--------+---------------------+------------------+---------------------+--------------------------------+---------------------+| 736352 | 2020-06-29 11:21:54 | NULL | 794751 | delete from S_VENUS_ACC_MASTER | READ COMMITTED |+--------+---------------------+------------------+---------------------+--------------------------------+---------------------+1 row in set (0.00 sec)
mysql> show processlist;+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+--------------------------------+| Id | User | Host | db | Command | Time | State | Info |+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+--------------------------------+| 11 | event_scheduler | localhost | NULL | Daemon | 2136775 | Waiting on empty queue | NULL || 778127 | system user | | NULL | Connect | 44730 | Waiting for master to send event | NULL || 794126 | root | localhost | information_schema | Query | 0 | starting | show processlist || 794691 | root | localhost | NULL | Sleep | 35 | | NULL || 794701 | mydata_repl | 133.96.30.143:45270 | NULL | Binlog Dump | 186 | Master has sent all binlog to slave; waiting for more updates | NULL || 794750 | system user | | NULL | Connect | 0 | Waiting for Slave Workers to free pending events | NULL || 794751 | system user | | NULL | Connect | 1211247 | Executing event | delete from S_VENUS_ACC_MASTER || 794752 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794753 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794754 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794755 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794756 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794757 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794758 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794759 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794760 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794761 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794762 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794763 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794764 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794765 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794766 | system user | | NULL | Connect | 49 | Waiting for an event from Coordinator | NULL || 794781 | mydata_ha | 133.96.30.141:58140 | NULL | Sleep | 1 | | NULL || 794782 | mydata_ha | 133.96.30.141:58142 | NULL | Sleep | 1 | | NULL || 794783 | mydata_ha | 133.96.30.141:58144 | NULL | Sleep | 1 | | NULL |+--------+-----------------+---------------------+--------------------+-------------+---------+---------------------------------------------------------------+--------------------------------+25 rows in set (0.00 sec)
以上是关于记录一则mysql主从延迟故障log的主要内容，如果未能解决你的问题，请参考以下文章
MySQL DBA日常故障处理----主从延迟故障(171020)