mysql多源复制(多主一从)配置

Posted 奋斗的一线码农

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了mysql多源复制(多主一从)配置相关的知识,希望对你有一定的参考价值。

mysql多源复制(多主一从)配置

应用场景

  • 数据汇总,可将多个主数据库同步汇总到一个从数据库中,方便数据统计分析。
  • 读写分离,从库只用于查询,提高数据库整体性能。

1.1.主库配置

my.cnf

#确保唯一

server-id=1

#作为Master要开启binlog

log-bin=mysql-bin

#binlog format有三种形式:Statement、Mixed、Row,默认设置为mixed

binlog-format=Row

#需要同步的库,不指定默认同步全部库

binlog-do-db=hi_db

#不需要同步的库

binlog-ignore-db=test

binlog-ignore-db=mysql

binlog-ignore-db=information_schema

binlog-ignore-db=performance_schema

binlog-ignore-db=sys

#这个比较重要,直接影响同步的性能 延时等问题.mysql5.7多源复制参数,5.7之前版本可不加

sync_binlog=1

auto_increment_increment=2

auto_increment_offset=1

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

#GTID模式

gtid-mode=on

enforce-gtid-consistency

1.2创建主库授权从库同步的用户

mysql>grant replication slave on *.* to ‘slave‘@‘192.168.100.%‘ identified by ‘308731044‘;

replication slave:远程用户备份权限

*.*:第一个星号代表库,第二个星号代表数据库里的表。可指定库和表

‘slave‘@‘192.168.100.%‘:@前为用户名,@后为授权的IP段(就是允许那些IP使用这个账号权限访问)

‘308731044‘:远程备份用户密码

1.3刷新设置(修改的内容刷新到数据库配置里 )

mysql>flush privileges;

查看和删除授权授权用户(删除不使用的账号)

log_bin是否开启

mysql> show variables like ‘log_bin‘;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| log_bin       | ON    |
+---------------+-------+
1 row in set

查看master状态

mysql> show master status G
*************************** 1. row ***************************
             File: mysql-bin.000003
         Position: 438
     Binlog_Do_DB: test3
 Binlog_Ignore_DB: mysql,information_schema,performation_schema,sys

1.4从库配置

#确保唯一

server-id = 3

#复制的库,不指定默认备份全部库

replicate-do-db = master1

replicate-do-db = master2

#不复制的库

replicate-ignore-db = mysql

replicate-ignore-db = information_schema

replicate-ignore-db = performance_schema

replicate-ignore-db = sys

#binlog日志设置

relay-log = /home/mysql/data/

mysqld-relay-bin

log-slave-updates = ON

slave-parallel-type=LOGICAL_CLOCK

relay_log_recovery=ON

#超时

slave_net_timeout = 30

#复制并发数设置

slave_parallel_workers = 16

#从库复制跳过错误

slave-skip-errors = 1062,1053,1146,1213,1264,1205,1396

sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

#mysql5.7多源复制必须添加的参数(不加报错),5.7版本之前不用加

master_info_repository=TABLE

relay_log_info_repository=TABLE

#GTID模式(使用GTID就可以不用记录备份位置点)

gtid-mode=on

enforce-gtid-consistency

1.5.从库配置用户及binlog位置信息(推荐使用GTID模式)

#GTID模式配置

CHANGE MASTER TO MASTER_HOST=‘192.168.1.131‘, MASTER_USER=‘slave‘, MASTER_PORT=3451, MASTER_PASSWORD=‘xxxxxx‘, MASTER_AUTO_POSITION = 1 FOR CHANNEL ‘master-1‘;

主库配置可以使用到多个主库上,从库添加相应的配置即可(GTID模式配置)

 

#Binlog模式

CHANGE MASTER TO
MASTER_HOST=‘10.60.1.126‘,
MASTER_PORT=3306,
MASTER_USER=‘root‘,
MASTER_PASSWORD=‘123456‘,
MASTER_LOG_FILE=‘mysql-bin.000002‘,
MASTER_LOG_POS=154
for channel ‘126‘;

CHANGE MASTER TO
MASTER_HOST=‘10.60.1.127‘,
MASTER_PORT=3306,
MASTER_USER=‘root‘,
MASTER_PASSWORD=‘123456‘,
MASTER_LOG_FILE=‘mysql-bin.000003‘,
MASTER_LOG_POS=154
for channel ‘127‘;

stop slave;     //停止同步
start slave;     //开始同步
//必须和【主库】的信息匹配。
CHANGE MASTER TO
MASTER_HOST=‘192.168.10.212‘,     //主库IP
MASTER_PORT=3600,                       //主库端口
MASTER_USER=‘root‘,                     //访问主库且有同步复制权限的用户
MASTER_PASSWORD=‘123456‘,      //登录密码
//【关键处】从主库的该log_bin文件开始读取同步信息,主库show master status返回结果
MASTER_LOG_FILE=‘mysql-bin.000003‘,
//【关键处】从文件中指定位置开始读取,主库show master status返回结果
MASTER_LOG_POS=438
for channel ‘126‘;            //定义通道名称

查看同步状态

mysql> show slave status G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.10.212
                  Master_User: slave
                  Master_Port: 4300
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000003
          Read_Master_Log_Pos: 1860
               Relay_Log_File: 0ad84f013600-relay-bin-300.000002
                Relay_Log_Pos: 1742
        Relay_Master_Log_File: mysql-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 1860
              Relay_Log_Space: 1960
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 300
                  Master_UUID: 4efae154-ebd4-11e8-bf7d-0242ac110007
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 300
           Master_TLS_Version: 
*************************** 2. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.10.212
                  Master_User: slave
                  Master_Port: 4400
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000003
          Read_Master_Log_Pos: 2461
               Relay_Log_File: 0ad84f013600-relay-bin-400.000002
                Relay_Log_Pos: 1187
        Relay_Master_Log_File: mysql-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 2461
              Relay_Log_Space: 1405
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 400
                  Master_UUID: 8fb0aa81-ebd4-11e8-ac5b-0242ac110009
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 400
           Master_TLS_Version: 
*************************** 3. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 192.168.10.212
                  Master_User: slave
                  Master_Port: 4500
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000003
          Read_Master_Log_Pos: 2151
               Relay_Log_File: 0ad84f013600-relay-bin-500.000002
                Relay_Log_Pos: 626
        Relay_Master_Log_File: mysql-bin.000003
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 2151
              Relay_Log_Space: 844
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 500
                  Master_UUID: b87704d2-ebd5-11e8-a90c-0242ac11000a
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 500
           Master_TLS_Version: 
3 rows in set (0.00 sec)

可以看见设置三个的主从同步通道的所有状态信息。
只有【Slave_IO_Running】和【Slave_SQL_Running】都是Yes,则同步是正常的。
如果是No或者Connecting都不行,可查看mysql-error.log,以排查问题。

mysql> show variables like ‘log_error%‘;
+---------------------+--------+
| Variable_name       | Value  |
+---------------------+--------+
| log_error           | stderr |
| log_error_verbosity | 3      |
+---------------------+--------+
2 rows in set

配置完成,则【从库128】开始自动同步。

若需要单独启动或停止某个同步通道,可使用如下命令:
start slave for channel ‘126‘;     //启动名称为300的同步通道
stop slave for channel ‘127‘;     //停止名称为300的同步通道

验证数据同步

建库

使用root账号登录【主库300】,创建test3数据库 

mysql> CREATE DATABASE test3;
Query OK, 1 row affected (0.00 sec)

mysql> USE test3;
Database changed

建表

在【主库300】中创建user表

CREATE TABLE `user` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `name` varchar(20) COLLATE utf8mb4_unicode_ci NOT NULL,
  `age` tinyint(3) unsigned NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

新增

在【主库300】中向user表插入一条数据:

mysql> use test3;
Database changed
mysql> INSERT INTO user (id, name, age) VALUES (300, ‘Tom‘, 18);
Database changed
mysql> SELECT * FROM user;
+-----+------+-----+
| id  | name | age |
+-----+------+-----+
| 300 | Tom  |  18 |
+-----+------+-----+
1 row in set (0.00 sec)

在【从库10345】中查询user表数据:

mysql> use test3;
Database changed
mysql> SELECT * FROM user;
+-----+------+-----+
| id  | name | age |
+-----+------+-----+
| 300 | Tom  |  18 |
+-----+------+-----+
1 row in set (0.00 sec)

新增记录同步成功。

更新

在【主库126】中修改刚才插入的数据:

mysql> UPDATE user SET name=‘Peter‘ where id=300;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> select * from user;
+-----+-------+-----+
| id  | name  | age |
+-----+-------+-----+
| 300 | Peter |  18 |
+-----+-------+-----+
1 row in set (0.00 sec)

在【从库10345】中查询user表数据:

mysql> select * from user;
+-----+-------+-----+
| id  | name  | age |
+-----+-------+-----+
| 300 | Peter |  18 |
+-----+-------+-----+
1 row in set (0.00 sec)

更新记录同步成功。

删除

在【主库126】中删除刚才更新的数据:

mysql> DELETE FROM user WHERE id=300;
Query OK, 1 row affected (0.00 sec)

mysql> select * from user;
Empty set (0.00 sec)

在【从库10345】中查询user表数据:

mysql> select * from user;
Empty set (0.00 sec)

删除记录同步成功。
注:【主库126】、【主库127】的验证操作与上述类似。

补充:

      • 如果【主服务器】重启mysql服务,【从服务器】会等待与【主服务器】重连。当主服务器恢复正常后,从服务器会自动重新连接上主服务器,并正常同步数据。
      • 如果某段时间内,【从数据库】服务器异常导致同步中断(可能是同步点位置不匹配),可以尝试以下恢复方法:进入【主数据库】服务器(正常),在bin-log中找到【从数据库】出错前的position,然后在【从数据库】上执行change master,将master_log_file和master_log_pos重新指定后,开始同步。 

 

④ 相关操作:

查看单个channel的状态:

show slave status for channel ‘t10‘G 

停止单个channel的同步:

stop slave for channel ‘t10‘;

开启单个channel的同步:

start slave for channel ‘t10‘;

重置单个channel:

reset slave all for channel ‘t10‘;

查看所有channel:

show slave statusG

停止所有channel:

stop slave;

开启所有channel:

start slave;

跳过一个channel的报错(类似MariaDB的default_master_connection):

处理方法:先停止所有的channel,再执行 sql_slave_skip_counter,接着开启报错的channel,最后开启所有的channel。

技术图片
一:
#stop all slaves
stop slave;

# set skip counter
set global sql_slave_skip_counter=1;

# start slave that shall skip one entry
start slave for channel ‘t10‘;

set global sql_slave_skip_counter=0;

# start all other slaves
start slave; 

二:
也可以直接停掉错误的channel,再skip:
stop slave for channel ‘t10‘;

set global sql_slave_skip_counter=1;

start slave for channel ‘t10‘;
技术图片

⑤ 监控系统库performance_schema增加了一些replication的监控表:

⑤ 监控系统库performance_schema增加了一些replication的监控表:

技术图片
mysql> show tables like ‘replicat%‘;
+-------------------------------------------+
| Tables_in_performance_schema (replicat%)  |
+-------------------------------------------+
| replication_applier_configuration         |###查看各个channel是否配置了复制延迟
| replication_applier_status                |###查看各个channel是否复制正常(service_state)以及事务重连的次数
| replication_applier_status_by_coordinator |###查看各个channel是否复制正常,以及复制错误的code、message和时间
| replication_applier_status_by_worker      |###查看各个channel是否复制正常,以及并行复制work号,复制错误的code、SQL和时间
| replication_connection_configuration      |###查看各个channel的连接配置信息:host、port、user、auto_position等
| replication_connection_status             |###查看各个channel的连接信息
| replication_group_member_stats            |###
| replication_group_members                 |###
+-------------------------------------------+
技术图片

...

2,在线调整Replication Filter

在上面搭建的主从基础上,进行过滤规则的添加,比如需要过滤dba_test数据库:

技术图片
先关闭sql线程,要是在多源复制中,是关闭所有channel的sql thread。
mysql> stop slave sql_thread;
Query OK, 0 rows affected (0.01 sec)

#过滤1个库
mysql> CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB=(dba_test);

#过滤2个库
mysql> CHANGE REPLICATION FILTER REPLICATE_IGNORE_DB=(dba_test1,dba_test);
Query OK, 0 rows affected (0.00 sec)

mysql> start slave sql_thread;
Query OK, 0 rows affected (0.04 sec)
技术图片

通过show slave status 查看:

              Replicate_Do_DB: 
          Replicate_Ignore_DB: dba_test1,dba_test
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 

比如设置同步dba_test2库中t1开头的表

技术图片
mysql> stop slave sql_thread;
Query OK, 0 rows affected (0.01 sec)

mysql> CHANGE REPLICATION FILTER REPLICATE_WILD_DO_TABLE =(‘dba_test2.t1%‘);
Query OK, 0 rows affected (0.00 sec)

mysql> start slave sql_thread;
Query OK, 0 rows affected (0.04 sec)
技术图片

还原成默认值,即设置成空():

技术图片
mysql> stop slave sql_thread;
Query OK, 0 rows affected (0.01 sec)

mysql> CHANGE REPLICATION FILTER REPLICATE_WILD_DO_TABLE=();
Query OK, 0 rows affected (0.00 sec)

mysql> CHANGE REPLICATION FILTER Replicate_Ignore_DB=();
Query OK, 0 rows affected (0.00 sec)

mysql> CHANGE REPLICATION FILTER Replicate_Wild_Ignore_Table=();
Query OK, 0 rows affected (0.00 sec)

mysql> start slave sql_thread;
Query OK, 0 rows affected (0.04 sec)
技术图片

用红色字体标记的这个参数就是设置在配置文件的参数,如上面的几个参数既可以在命令行里执行(5.7)也可以在配置文件里添加。注意一点是在线执行完后,一定要在配置文件里写,以免重启后失效。

以上是关于mysql多源复制(多主一从)配置的主要内容,如果未能解决你的问题,请参考以下文章

MySQL 5.7的多源复制

MySQL多源复制

MySQL 5.7 Replication 相关新功能说明

MySQL 5.7 多主一丛同步的数据库配置(将多处数据源合并到一点)

mysql 5.7.16多源复制

Mysql 多主一从数据备份