mysql主从之多线程复制
Posted zyxnhr
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了mysql主从之多线程复制相关的知识，希望对你有一定的参考价值。
多线程复制
mysql 主从复制原理：
1. master 节点上的binlogdump 线程，在slave 与其正常连接的情况下，将binlog 发送到slave 上。
2. slave 节点的I/O Thread ，通过读取master 节点binlog 日志名称以及偏移量信息将其拷贝到本地relay log 日志文件。
3. slave 节点的SQL Thread，该线程读取relay log 日志信息，将在master 节点上提交的事务在本地回放，达到与主库数据保持一致的目的。
MySQL5.5 及以前的复制
一般主从复制有三个线程且都是单线程：
Binlog Dump（主） ‐‐> IO Thread（从） ‐‐> SQL Thread（从）。
而master 这边是通过并发线程提交，事务通过LSN 写入binlog；但是Slave 只有一个IO 线程和SQL 线程，是单线程，所以在业务大的情况下就很容易造成主从延时.
如果在MySQL 5.6 版本开启并行复制功能（slave_parallel_workers > 0），那么SQL 线程就变为了coordinator 线程，coordinator 线程主要负责以下两部分内容：
Coordinator+worker（多个）
若判断可以并行执行，那么选择worker 线程执行事务的二进制日志。
若判断不可以并行执行，如该操作是DDL，亦或者是事务跨schema 操作，则等待所有的worker 线程执行完成之后，再执行当前的日志。
这意味着coordinator 线程并不是仅将日志发送给worker 线程，自己也可以回放日志，但是所有可以并行的操作交付由worker 线程完成。
上述机制实现的基于schema 的并行复制，存在的问题是：
这样设计的并行复制效果并不高，如果用户实例仅有一个库，那么就无法实现并行回放，甚至性能会比原来的单线程更差，而单库多表是比多库多表更为常见的一种情形。
MySQL5.7 的MTS（Enhanced Muti‐threadedslaves）MySQL 5.7 引入了新的机制来实现并行复制，不再有基于库的并行复制限制，主要思想就是slave 服务器的回放与主机是一致的，即master 服务器上是怎么并行执行，slave 上就怎样进行并行回放。
mysql v5.7.2 进行了优化，增加了参数slave_parallel_type，参数有两个选项：
LOGICAL_CLOCK：基于逻辑时钟 ，可以在一个DATABASE 中并发执行relay log 事务DATABASE： 基于数据库，v5.6 默认是这个参数，改参数每个库只能一个线程；
slave‐parallel‐type，其可以配置的值有：
DATABASE：默认值，基于库的并行复制方式
LOGICAL_CLOCK：基于组提交的并行复制方式
从库:
mysql> show variables like ‘slave_parallel%‘; +------------------------+----------+ | Variable_name | Value | +------------------------+----------+ | slave_parallel_type | DATABASE | | slave_parallel_workers | 0 | +------------------------+----------+ mysql> stop slave; mysql> set global slave_parallel_type=‘LOGICAL_CLOCK‘; Query OK, 0 rows affected (0.00 sec) mysql> set global slave_parallel_type=‘LOGICAL_CLOCK‘; Query OK, 0 rows affected (0.00 sec) mysql> set global slave_parallel_workers=2; Query OK, 0 rows affected (0.00 sec) mysql> start slave; Query OK, 0 rows affected (0.01 sec) mysql> show processlist; +-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+ | 26077 | root | localhost | NULL | Query | 0 | starting | show processlist | | 26141 | system user | | NULL | Connect | 40 | Waiting for master to send event | NULL | | 26142 | system user | | NULL | Connect | 40 | Slave has read all relay log; waiting for more updates | NULL | | 26143 | system user | | NULL | Connect | 40 | Waiting for an event from Coordinator | NULL | | 26144 | system user | | NULL | Connect | 40 | Waiting for an event from Coordinator | NULL | +-------+-------------+-----------+------+---------+------+--------------------------------------------------------+------------------+ mysql> show variables like ‘slave_parallel%‘; +------------------------+---------------+ | Variable_name | Value | +------------------------+---------------+ | slave_parallel_type | LOGICAL_CLOCK | | slave_parallel_workers | 2 | +------------------------+---------------+
master_info_repository=TABLE （开启MTS 功能后，会频繁更新master.info，设置为TABLE 减小开销）relay_log_info_repository=TABLE
slave_master_info 记录了首次同步master 的位置relay_log_recovery=ON （slave IO 线程crash，如果relay‐log损坏，则自动放弃所有未执行的relay‐log，重新从master 上获取日志，保证relay‐log 的完整性）
slave_preserve_commit_order=ON （保证提交的顺序性）在slave 上应用事务的顺序是无序的，和relay log 中记录的事务顺序不一样，这样数据一致性是无法保证的，为了保证事务是按照relay log 中记录的顺序来回放，就需要开启参数slave_preserve_commit_order。
虽然mysql5.7 添加MTS 后，虽然slave 可以并行应用relay log，但commit 部分仍然是顺序提交，其中可能会有等待的情况。
优化参数
[mysqld]
bind-address=0.0.0.0
port=3306
datadir=/data/mysql
socket=/data/mysql/mysql.sock
user=mysql
skip-name-resolve
slow_query_log=on
long_query_time=1
slow_query_log_file=/data/mysql/mysql-slow.log
innodb-file-per-table=1
innodb_flush_log_at_trx_commit = 2
log_warnings = 1
connect_timeout = 60
net_read_timeout = 120
performance_schema_max_table_instances = 400
server-id = 2
relay-log = relay-log
relay-log-index = relay-log.index
slave_parallel_type=LOGICAL_CLOCK
slave_parallel_workers=2
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON
slave_preserve_commit_order=ON
log_bin=mysql‐bin
log_slave_updates=ON
[mysqld_safe]
log-error=/data/mysql/mysqld.log
pid-file=/data/mysql/mysqld.pid
重启mysqld进入数据库
mysql> show variables like ‘%slave_parallel_type%‘; +---------------------+---------------+ | Variable_name | Value | +---------------------+---------------+ | slave_parallel_type | LOGICAL_CLOCK | +---------------------+---------------+ mysql> show variables like ‘%slave_parallel_worker%‘; +------------------------+-------+ | Variable_name | Value | +------------------------+-------+ | slave_parallel_workers | 2 | +------------------------+-------+ mysql> show variables like ‘%master_info_repository%‘; +------------------------+-------+ | Variable_name | Value | +------------------------+-------+ | master_info_repository | TABLE | +------------------------+-------+ mysql> set global relay_log_info_repository=‘TABLE‘; Query OK, 0 rows affected (0.00 sec) mysql> show variables like ‘%relay_log_info_repository%‘; +---------------------------+-------+ | Variable_name | Value | +---------------------------+-------+ | relay_log_info_repository | TABLE | +---------------------------+-------+ mysql> show variables like ‘%relay_log_recovery%‘; +--------------------+-------+ | Variable_name | Value | +--------------------+-------+ | relay_log_recovery | ON | +--------------------+-------+ 1 row in set (0.00 sec) mysql> show variables like ‘%slave_preserve_commit_order%‘; +-----------------------------+-------+ | Variable_name | Value | +-----------------------------+-------+ | slave_preserve_commit_order | ON | +-----------------------------+-------+
开启slave
mysql> start slave; Query OK, 0 rows affected (0.00 sec) mysql> show slave status\\G; *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 192.168.132.121 Master_User: replication Master_Port: 3306 Connect_Retry: 60 Master_Log_File: master-bin.000002 Read_Master_Log_Pos: 40870 Relay_Log_File: relay-log.000010 Relay_Log_Pos: 321 Relay_Master_Log_File: master-bin.000002 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 40870 Relay_Log_Space: 689 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 6b00724f-a094-11e9-8f47-000c2991dd19 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version:
主库写入数据
[[email protected] mysql]#mysql -uroot -p123456 -e ‘use darren;insert into test values (1); [[email protected] mysql]#mysql -uroot -p123456 -e ‘use darren;insert into test values (1); [[email protected] mysql]#mysql -uroot -p123456 mysql> mysql> show master status; +-------------------+----------+--------------+------------------+-------------------+ | File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | +-------------------+----------+--------------+------------------+-------------------+ | master-bin.000002 | 41388 | | | | +-------------------+----------+--------------+------------------+-------------------+
从库
mysql> use mysql; Database changed mysql> show tables; +---------------------------+ | Tables_in_mysql | +---------------------------+ | columns_priv | | db | | engine_cost | | event | | func | | general_log | | gtid_executed | | help_category | | help_keyword | | help_relation | | help_topic | | innodb_index_stats | | innodb_table_stats | | ndb_binlog_index | | plugin | | proc | | procs_priv | | proxies_priv | | server_cost | | servers | | slave_master_info | | slave_relay_log_info | | slave_worker_info | | slow_log | | tables_priv | | time_zone | | time_zone_leap_second | | time_zone_name | | time_zone_transition | | time_zone_transition_type | | user | +---------------------------+ mysql> select * from slave_master_info; +-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+ | Number_of_lines | Master_log_name | Master_log_pos | Host | User_name | User_password | Port | Connect_retry | Enabled_ssl | Ssl_ca | Ssl_capath | Ssl_cert | Ssl_cipher | Ssl_key | Ssl_verify_server_cert | Heartbeat | Bind | Ignored_server_ids | Uuid | Retry_count | Ssl_crl | Ssl_crlpath | Enabled_auto_position | Channel_name | Tls_version | +-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+ | 25 | master-bin.000002 | 40870 | 192.168.132.121 | replication | 1234567 | 3306 | 60 | 0 | | | | | | 0 | 30 | | 0 | 6b00724f-a094-11e9-8f47-000c2991dd19 | 86400 | | | 0 | | | +-----------------+-------------------+----------------+-----------------+-------------+---------------+------+---------------+-------------+--------+------------+----------+------------+---------+------------------------+-----------+------+--------------------+--------------------------------------+-------------+---------+-------------+-----------------------+--------------+-------------+ mysql> select * from slave_relay_log_info; +-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+ | Number_of_lines | Relay_log_name | Relay_log_pos | Master_log_name | Master_log_pos | Sql_delay | Number_of_workers | Id | Channel_name | +-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+ | 7 | ./relay-log.000010 | 839 | master-bin.000002 | 41388 | 0 | 2 | 1 | | +-----------------+--------------------+---------------+-------------------+----------------+-----------+-------------------+----+--------------+
查看线程
+----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+ | Id | User | Host | db | Command | Time | State | Info | +----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+ | 7 | root | localhost | mysql | Query | 0 | starting | show processlist | | 36 | system user | | NULL | Connect | 762 | Waiting for master to send event | NULL | | 37 | system user | | NULL | Connect | 567 | Slave has read all relay log; waiting for more updates | NULL | | 38 | system user | | NULL | Connect | 567 | Waiting for an event from Coordinator | NULL | | 39 | system user | | NULL | Connect | 762 | Waiting for an event from Coordinator | NULL | +----+-------------+-----------+-------+---------+------+--------------------------------------------------------+-------------------+
以上是关于mysql主从之多线程复制的主要内容，如果未能解决你的问题，请参考以下文章