MHA

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了MHA相关的知识,希望对你有一定的参考价值。

MHA原理:

(1)一主三从架构,主库挂了,但主库能被从库ssh上去的情况下,MHA从三个从库中选择同步最接近的作为新主,然后新主和s2,s3都ssh到原主上通过binlog补上还没有同步的数据,io_thread读取到binlog位置,传到save_binary_logs,然后回放,达到s1,s2,s3和原主一致。

(2)主库无法ssh上去的情况下,即主库的系统无法连接,假设有binlog server ,利用binlog补数据,和前面情况一样;

如果没有binlog server,假设s1靠前,s2和s3会通过relay-log和s1同步(所以从库的relay不能自动清除参数relay_log_purge = 0) ,提升s1为主,把s2和s3作为s1的从;

(3)5.6之后,如果有一主三从的架构,使用GTID复制,keepalived+一主两从,并且其中一个从库使用增强半同步复制,这样完全可以取代MHA,这就是为什么MHA代码为什么不更新,out的原因。


实验环境:

OS:CentOS release 6.6 (Final)

数据库:mysql 5.7.16

master :192.168.91.23

slave1 :192.168.91.22

slave2:192.168.91.21

VIP:192.168.91.200


master && masterha_managerDarren1192.168.91.23 
slave1Darren2192.168.91.22
slave2Darren3192.168.91.21


实验架构图:

技术分享

提前做好一主两从传统的复制(这里不再演示)

提前安装依赖包:(一般yum源可能没有这些依赖包,可以安装epel)

[[email protected] tools]# rpm -ivh epel-release-latest-6.noarch.rpm

[[email protected] tools]# rpm -ivh epel-release-latest-6.noarch.rpm

[[email protected] tools]# rpm -ivh epel-release-latest-6.noarch.rpm


下载软件链接:

http://code.google.com/p/mysql-master-ha/wiki


#master上安装manager和node:

[[email protected] tools]# ls

mha4mysql-manager-0.56-0.el6.noarch.rpm 

mha4mysql-node-0.56-0.el6.noarch.rpm


[[email protected] tools]# yum localinstall mha4mysql*


[[email protected] mysql]# rpm -qa |grep mha

mha4mysql-node-0.56-0.el6.noarch

mha4mysql-manager-0.56-0.el6.noarch


#如果yum也无法安装,那就先安装依赖包,再安装rpm包:

yum install perl-DBD-MySQL  perl-Config-Tiny perl-Log-Dispatch perl-Parallel-ForkManager -y

rpm -ivh mha4mysql-node-0.56-0.el6.noarch.rpm

rpm -ivh mha4mysql-manager-0.56-0.el6.noarch.rpm


#slave1和slave2在从库上安装node软件:

[[email protected] app1]# yum localinstall mha4mysql-node-0.56-0.el6.noarch.rpm

[[email protected] app1]# yum localinstall mha4mysql-node-0.56-0.el6.noarch.rpm


物理机器之间信任机制建立


#三台机器同样配置:

[[email protected] tools]# vim  /etc/hosts

192.168.91.23 Darren1

192.168.91.22 Darren2

192.168.91.21 Darren3


在master上面操作:

[[email protected] ]# cd  /root/.ssh

[[email protected] .ssh]# ssh-keygen   #一路回车

[[email protected] .ssh]# cat /etc/ssh/sshd_config |grep authorized

#AuthorizedKeysFile     .ssh/authorized_keys


#把生成的秘钥文件重定向到authorized_keys文件中:

[[email protected] .ssh]# cat id_rsa.pub >>authorized_keys

[[email protected] .ssh]# chmod 600 *


#将创建的文件拷贝到slave1和slave2上面:

[[email protected] .ssh]# scp * 192.168.91.22:/root/.ssh/

[[email protected] .ssh]# scp * 192.168.91.21:/root/.ssh/


#在slave1和slave2上面操作:

[[email protected] .ssh]#chmod 600  /root/.ssh/*

[[email protected] .ssh]#chmod 600  /root/.ssh/*


#测试互信是否成功:

[[email protected] ~]# ssh Darren2 hostname

[[email protected] ~]# ssh Darren3 hostname

[[email protected] ~]# ssh Darren1 hostname

[[email protected] ~]# ssh Darren3 hostname

[[email protected] ~]# ssh Darren1 hostname

[[email protected] ~]# ssh Darren2 hostname


#创建masterha日志文件目录:

[[email protected] masterha]#mkdir -p /var/log/masterha/app1


#在master上创建目录及配置文件:

[[email protected] ~]# mkdir -p /etc/masterha

[[email protected] ~]# cd /etc/masterha/


#上传五个文件,文件及脚本内容在文章末尾处贴出来:

[[email protected] masterha]# ls

app1.conf  drop_vip.sh  init_vip.sh  masterha_default.conf  master_ip_failover  master_ip_online_change


[[email protected] masterha]# chmod +x drop_vip.sh init_vip.sh master_ip_failover master_ip_online_change


 ------------------------------------------------------------

[[email protected] masterha]# cat app1.conf

[server default]

#mha manager工作目录

manager_workdir =     /var/log/masterha/app1

manager_log     =    /var/log/masterha/app1/app1.log

remote_workdir  =   /var/log/masterha/app1

[server1]

hostname=192.168.91.23

master_binlog_dir = /data/mysql/mysql3306/logs

candidate_master  = 1

check_repl_delay  = 0     #用防止master故障时,切换时slave有延迟,卡在那里切不过来。

[server2]

hostname=192.168.91.22

master_binlog_dir=/data/mysql/mysql3306/logs

candidate_master=1

check_repl_delay=0

[server3]

hostname=192.168.91.21

master_binlog_dir=/data/mysql/mysql3306/logs

candidate_master=1

check_repl_delay=0


[[email protected] masterha]# cat masterha_default.conf

[server default]

#MySQL的用户和密码

user=admin

password = 147258

#系统ssh用户

ssh_user=root

#复制用户

repl_user=repl

repl_password= 147258

#监控

ping_interval=1

#shutdown_script=""

#切换调用的脚本

master_ip_failover_script= /etc/masterha/master_ip_failover

master_ip_online_change_script= /etc/masterha/master_ip_online_change

-------------------------------------------


#分别传到slave1和slave2上相同位置:

[[email protected] masterha]# scp -r /etc/masterha 192.168.91.22:/etc/

[[email protected] masterha]# scp -r /etc/masterha 192.168.91.21:/etc/


#创建一个admin用户,用于启动masterha_manager :

[email protected] [(none)]>create user [email protected]‘192.168.91.%‘ identified by ‘147258‘;

[email protected] [(none)]>grant all on *.* to ‘admin‘@‘192.168.91.%‘;   


#启动VIP:

[[email protected] masterha]# sh init_vip.sh


#节点之间的主从关系检测:

[[email protected] masterha]# masterha_check_repl --global_conf=/etc/masterha/masterha_default.conf --conf=/etc/masterha/app1.conf

最后输出:MySQL Replication Health is OK.表示检测通过


#节点直接的互信检测:

[[email protected] masterha]# masterha_check_ssh --global_conf=/etc/masterha/masterha_default.conf --conf=/etc/masterha/app1.conf

Thu May 11 15:24:55 2017 - [info] All SSH connection tests passed successfully.


#启动masterha:

[[email protected] app1]# nohup masterha_manager --global_conf=/etc/masterha/masterha_default.conf --conf=/etc/masterha/app1.conf > /tmp/masterha_manager.log 2>&1 &


#查看masterha状态:

[[email protected] app1]# masterha_check_status --global_conf=/etc/masterha/masterha_default.conf --conf=/etc/masterha/app1.conf

app1 (pid:86957) is running(0:PING_OK), master:192.168.91.23


[[email protected] app1]# ps -ef |grep master

root       1371      1  0 May04 ?        00:00:02 /usr/libexec/postfix/master

root      78618  67225  0 08:02 pts/0    00:00:00 perl /usr/bin/masterha_manager --global_conf=/etc/masterha/masterha_default.conf --conf=/etc/masterha/app1.conf

root      79204  67246  0 08:07 pts/2    00:00:00 grep master


#通过日志文件查看启动过程:

-----------------------------------------------

[[email protected] app1]# tail -f app1.log

Thu May 11 15:47:15 2017 - [info] MHA::MasterMonitor version 0.56.

Thu May 11 15:47:16 2017 - [info] GTID failover mode = 0

Thu May 11 15:47:16 2017 - [info] Dead Servers:

Thu May 11 15:47:16 2017 - [info] Alive Servers: --罗列mha中的所有节点

Thu May 11 15:47:16 2017 - [info]   192.168.91.23(192.168.91.23:3306)

Thu May 11 15:47:16 2017 - [info]   192.168.91.22(192.168.91.22:3306)

Thu May 11 15:47:16 2017 - [info] Alive Slaves:  --活跃的从库

Thu May 11 15:47:16 2017 - [info]   192.168.91.22(192.168.91.22:3306)  Version=5.7.16-log (oldest major version between slaves) log-bin:enabled

Thu May 11 15:47:16 2017 - [info]     Replicating from 192.168.91.23(192.168.91.23:3306)

Thu May 11 15:47:16 2017 - [info]     Primary candidate for the new Master (candidate_master is set)

Thu May 11 15:47:16 2017 - [info] Current Alive Master: 192.168.91.23(192.168.91.23:3306) --现在的主库

Thu May 11 15:47:16 2017 - [info] Checking slave configurations..

Thu May 11 15:47:16 2017 - [info]  read_only=1 is not set on slave 192.168.91.22(192.168.91.22:3306).

Thu May 11 15:47:16 2017 - [info] Checking replication filtering settings..

Thu May 11 15:47:16 2017 - [info]  binlog_do_db= , binlog_ignore_db=

Thu May 11 15:47:16 2017 - [info]  Replication filtering check ok.

Thu May 11 15:47:16 2017 - [info] GTID (with auto-pos) is not supported

Thu May 11 15:47:16 2017 - [info] Starting SSH connection tests..

Thu May 11 15:47:17 2017 - [info] All SSH connection tests passed successfully.

Thu May 11 15:47:17 2017 - [info] Checking MHA Node version..

Thu May 11 15:47:17 2017 - [info]  Version check ok.

Thu May 11 15:47:17 2017 - [info] Checking SSH publickey authentication settings on the current master..

Thu May 11 15:47:17 2017 - [info] HealthCheck: SSH to 192.168.91.23 is reachable.

Thu May 11 15:47:17 2017 - [info] Master MHA Node version is 0.56.

Thu May 11 15:47:17 2017 - [info] Checking recovery script configurations on 192.168.91.23(192.168.91.23:3306)..

save_binary_logs

#检查binlog部分,如果主库挂了,从库需要用到binlog补数据,这个命令是主库故障转移时候用到,这里是command=test是测试,没有实际操作。

Thu May 11 15:47:17 2017 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/mysql3306/logs --output_file=/var/log/masterha/app1/save_binary_logs_test --manager_version=0.56 --start_file=mysql-bin.000029

Thu May 11 15:47:17 2017 - [info]   Connecting to [email protected](192.168.91.23:22)..

  Creating /var/log/masterha/app1 if not exists..    ok.

  Checking output directory is accessible or not..

   ok.

  Binlog found at /data/mysql/mysql3306/logs, up to mysql-bin.000029

Thu May 11 15:47:17 2017 - [info] Binlog setting check done.


#查看从库的relay-log,使用到命令apply_diff_relay_logs,如果主库挂了,并且不能ssh上去也没有binlog server备份,那么选举一个从库作为新主,其他从库需要用到这里的relay-log补数据;

Thu May 11 15:47:17 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..

Thu May 11 15:47:17 2017 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user=‘admin‘ --slave_host=192.168.91.22 --slave_ip=192.168.91.22 --slave_port=3306 --workdir=/var/log/masterha/app1 --target_version=5.7.16-log --manager_version=0.56 --relay_dir=/data/mysql/mysql3306/data --current_relay_log=relay-bin.000001  --slave_pass=xxx

Thu May 11 15:47:17 2017 - [info]   Connecting to [email protected](192.168.91.22:22)..

  Checking slave recovery environment settings..

    Relay log found at /data/mysql/mysql3306/data, up to relay-bin.000002

    Temporary relay log file is /data/mysql/mysql3306/data/relay-bin.000002

    Testing mysql connection and privileges..mysql: [Warning] Using a password on the command line interface can be insecure.

 done.

    Testing mysqlbinlog output.. done.

    Cleaning up test file(s).. done.

Thu May 11 15:47:18 2017 - [info] Slaves settings check done.

Thu May 11 15:47:18 2017 - [info]

#现在的主库和从库

192.168.91.23(192.168.91.23:3306) (current master)

 +--192.168.91.22(192.168.91.22:3306)

Thu May 11 15:47:18 2017 - [info] Checking master_ip_failover_script status:

Thu May 11 15:47:18 2017 - [info]   /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.91.23 --orig_master_ip=192.168.91.23 --orig_master_port=3306

Thu May 11 15:47:18 2017 - [info]  OK.

Thu May 11 15:47:18 2017 - [warning] shutdown_script is not defined.

#每隔1秒ping一次

Thu May 11 15:47:18 2017 - [info] Set master ping interval 1 seconds.


Thu May 11 15:47:18 2017 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.

Thu May 11 15:47:18 2017 - [info] Starting ping health check on 192.168.91.23(192.168.91.23:3306)..

Thu May 11 15:47:18 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn‘t respond..

------------------------------------

#停止mha:

[[email protected] app1]# masterha_stop --global_conf=/etc/masterha/masterha_default.conf --conf=/etc/masterha/app1.conf      

Stopped app1 successfully.


当主库上的mysql挂了,这个时候会从slave1和slave2中选择一个作为新主,如果选择slave1作为新主,这个时候slave2会把slave1作为新主,以前挂掉的主等修复好之后可以作为slave1的从。


技术分享

模拟切换

[[email protected] app1]# /etc/init.d/mysqld stop

Shutting down MySQL............ SUCCESS!


#MHA日志目录下会生成两个文件,其中app1.failover.complete 这个文件下次启动mha之前必须删除或者移走,否则无法启动:

[[email protected] app1]# ll

total 24

-rw-r--r-- 1 root root     0 May 10 08:12 app1.failover.complete

-rw-r--r-- 1 root root 17416 May 10 08:12 app1.log

-rw-r--r-- 1 root root   234 May 10 08:12 saved_master_binlog_from_192.168.91.23_3306_20170510081201.binlog


#可以看到ip已经切换到从库上面:

[[email protected] masterha]# ip addr|grep 192

    inet 192.168.91.22/24 brd 192.168.91.255 scope global eth0

    inet 192.168.91.200/32 scope global eth0

    inet 192.168.91.100/32 scope global eth0


#slave1上的从库信息也被清除了:

[email protected] [(none)]>show slave status\G

Empty set (0.00 sec)


#slave2也把slave1作为了新主:

[email protected] [(none)]>show slave status\G

*************************** 1. row ***************************

               Slave_IO_State: Waiting for master to send event

                  Master_Host: 192.168.91.22

                  Master_User: repl


挂掉的主库怎么办?

可以作为新主的从库:

[[email protected] app1]# /etc/init.d/mysqld start


#在MHA日志中可以找到master_log_file和master_log_pos:

[[email protected] app1]# vim app1.log|grep CHANGE


[email protected] [(none)]>change master to

    master_host=‘192.168.91.22‘,

    master_port=3306,

    master_user=‘repl‘,

    master_password=‘147258‘,

    Master_Log_File=‘mysql-bin.000014‘,

    Master_Log_Pos=234;


#重新启动mha:

[[email protected] app1]# nohup masterha_manager --global_conf=/etc/masterha/masterha_default.conf --conf=/etc/masterha/app1.conf > /tmp/masterha_manager.log 2>&1 &


[[email protected] app1]# cat app1.master_status.health

94112   0:PING_OK       master:192.168.91.22


[[email protected] app1]# tail app1.log

Thu May 11 17:06:42 2017 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.

Thu May 11 17:06:42 2017 - [info] Starting ping health check on 192.168.91.22(192.168.91.22:3306)..  --应为这个时候22已经是主库了,所以这里一直ping22是否存在

Thu May 11 17:06:42 2017 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn‘t respond..


可能遇到的报错

(1)[[email protected] .ssh]# masterha_check_ssh --global_conf=/etc/masterha/masterha_default.conf --conf=/etc/masterha/app1.conf

......

Thu May 11 19:42:40 2017 - [info]   Connecting to [email protected](192.168.91.21:22)..

Can‘t exec "mysqlbinlog": No such file or directory at /usr/share/perl5/vendor_perl/MHA/BinlogManager.pm line 106.

或者执行同样的命令遇到这样的报错:

Testing mysql connection and privileges..sh: mysql: command not found

mysql command failed with rc 127:0!

解决方法:

需要做一个软连接

ln -s /usr/local/mysql/bin/mysqlbinlog /usr/bin/mysqlbinlog

ln -s /usr/local/mysql/bin/mysql /usr/bin/mysql     


(2)在启动masterha_manager的时候可能报错:

[[email protected] app1]# tail app1.log

Thu May 11 21:40:18 2017 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln309] Last failover was done at 2017/05/11 19:04:14. Current time is too early to do failover again. If you want to do failover, manually remove /var/log/masterha/app1/app1.failover.complete and run this script again.

Thu May 11 21:40:18 2017 - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR:  at /usr/bin/masterha_manager line 65

原因是没有删除app1.failover.complete,rm就好。


本文出自 “10979687” 博客,请务必保留此出处http://10989687.blog.51cto.com/10979687/1924835

以上是关于MHA的主要内容,如果未能解决你的问题,请参考以下文章

MHA高可用集群

MHA高可用群集

MHA高可用集群部署及故障切换

MHA 介绍

MHA你难道不了解一下?mysql数据库MHA高可用配置及故障切换!

MHA架构