MHA 安装及测试

Posted 2021-01-18

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了MHA 安装及测试相关的知识，希望对你有一定的参考价值。

准备工作

1.下载安装包
2.配置好主从

主：192.168.160.132
从：192.168.160.133 192.168.160.134
mha：192.168.160.135

GRANT SUPER, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO ‘replication_user‘@‘192.168.160.%‘ IDENTIFIED BY ‘123456‘;  #所有的mysql服务器都要创建复制账号
change master to master_host=‘192.168.160.132‘,master_user=‘replication_user‘, master_port=3388,master_password=‘123456‘,MASTER_PORT = 3388,MASTER_LOG_FILE = ‘bin.000004‘,MASTER_LOG_POS =722;  #160.133  160.134  两台机器

3.创建监控账号 #所有MySQL服务器

grant all privileges on *.* to ‘root‘@‘192.168.31.%‘ identified by ‘123‘;   #监控用户

4.配置SSH 无密码登录 ##配置所有的机器相互信任

 cd .ssh/
ssh-keygen #这里什么也不填，一路回车，这样保证无密码
cat id_rsa.pub >> authorized_keys # 重定向
scp .ssh/* 192.168.160.133:~/.ssh/    #cp文件到其他节点上
 scp .ssh/* 192.168.160.134:~/.ssh/
 scp .ssh/* 192.168.160.135:~/.ssh/
chmod 600 * # 授权ssh下的文件
#将每个主机的id_rsa.pub 公钥 加到其他主机的authorized_keys中

####安装步骤（rpm包安装）
1.所有节点包括manage，安装node节点

yum localinstall mha4mysql-node-0.57-0.el7.noarch.rpm
#安装完后，可以通过    which save_binary_logs 检测

2.给manager安装manager

yum localinstall mha4mysql-manager-0.57-0.el7.noarch.rpm
#安装过程中可能会有依赖包问题，按照报错信息安装依赖包就好
#masterha_check_ssh 检查MHA的SSH配置状况
#masterha_check_repl 检查MySQL复制状况
#masterha_manger 启动MHA
#masterha_check_status 检测当前MHA运行状态
#masterha_master_monitor 检测master是否宕机
#masterha_master_switch 控制故障转移（自动或者手动）
#masterha_conf_host 添加或删除配置的server信息

3.配置MHA

mkdir -p /etc/mha/{log,app1,conf,scritps}    #创建相对应的文件夹
vim /etc/mha/conf/app1.conf
[server default]
# 这边是 连接 MySQL 的账号与密码， 如果端口发生改变， 也要写上相应的端口, 默认为 3306
#port=3388
user=root
password=123456
ssh_user=root

# 这边复制账号
repl_user=replication_user
repl_password=123456
master_binlog_dir=/log/binlog/
master_ip_failover_script=/etc/mha/scripts/master_ip_failover_new
master_ip_online_change_script=/etc/mha/scripts/master_online_change_new
manager_workdir=/etc/mha/app1
manager_log=/etc/mha/log/manager.log

[server1]
hostname=192.168.160.132
candidate_master=1
port = 3388
master_binlog_dir=/log/binlog
[server2]
hostname=192.168.160.133
port = 3388
candidate_master=1
master_binlog_dir=/log/binlog
[server3]
hostname=192.168.160.134
port = 3306
candidate_master=1
master_binlog_dir=/log/binlog

[binlog1]
hostname=192.168.160.132
[binlog2]
hostname=192.168.160.133
[binlog3]
hostname=192.168.160.134

4.检测

[[email protected] opt]# masterha_check_repl --conf=/etc/mha/conf/app1.conf 
Thu Nov 8 03:11:53 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Nov 8 03:11:53 2018 - [info] Reading application default configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 03:11:53 2018 - [info] Reading server configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 03:11:53 2018 - [info] MHA::MasterMonitor version 0.57.
Thu Nov 8 03:11:53 2018 - [info] GTID failover mode = 0
Thu Nov 8 03:11:53 2018 - [info] Dead Servers:
Thu Nov 8 03:11:53 2018 - [info] Alive Servers:
Thu Nov 8 03:11:53 2018 - [info] 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 03:11:53 2018 - [info] 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 03:11:53 2018 - [info] 192.168.160.134(192.168.160.134:3306)
Thu Nov 8 03:11:53 2018 - [info] Alive Slaves:
Thu Nov 8 03:11:53 2018 - [info] 192.168.160.133(192.168.160.133:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 03:11:53 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 03:11:53 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 03:11:53 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 03:11:53 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 03:11:53 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 03:11:53 2018 - [info] Current Alive Master: 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 03:11:53 2018 - [info] Checking slave configurations..
Thu Nov 8 03:11:53 2018 - [info] read_only=1 is not set on slave 192.168.160.133(192.168.160.133:3388).
Thu Nov 8 03:11:53 2018 - [warning] relay_log_purge=0 is not set on slave 192.168.160.133(192.168.160.133:3388).
Thu Nov 8 03:11:53 2018 - [info] read_only=1 is not set on slave 192.168.160.134(192.168.160.134:3306).
Thu Nov 8 03:11:53 2018 - [warning] relay_log_purge=0 is not set on slave 192.168.160.134(192.168.160.134:3306).
Thu Nov 8 03:11:53 2018 - [info] Checking replication filtering settings..
Thu Nov 8 03:11:53 2018 - [info] binlog_do_db= , binlog_ignore_db= 
Thu Nov 8 03:11:53 2018 - [info] Replication filtering check ok.
Thu Nov 8 03:11:53 2018 - [info] GTID (with auto-pos) is not supported
Thu Nov 8 03:11:53 2018 - [info] Starting SSH connection tests..
Thu Nov 8 03:11:56 2018 - [info] All SSH connection tests passed successfully.
Thu Nov 8 03:11:56 2018 - [info] Checking MHA Node version..
Thu Nov 8 03:11:56 2018 - [info] Version check ok.
Thu Nov 8 03:11:56 2018 - [info] Checking SSH publickey authentication settings on the current master..
Thu Nov 8 03:11:57 2018 - [info] HealthCheck: SSH to 192.168.160.132 is reachable.
Thu Nov 8 03:11:57 2018 - [info] Master MHA Node version is 0.57.
Thu Nov 8 03:11:57 2018 - [info] Checking recovery script configurations on 192.168.160.132(192.168.160.132:3388)..
Thu Nov 8 03:11:57 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/log/binlog --output_file=/var/tmp/save_binary_logs_test --manager_version=0.57 --start_file=bin.000004 
Thu Nov 8 03:11:57 2018 - [info] Connecting to [email protected](192.168.160.132:22).. 
  Creating /var/tmp if not exists.. ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /log/binlog, up to bin.000004
Thu Nov 8 03:11:59 2018 - [info] Binlog setting check done.
Thu Nov 8 03:11:59 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Thu Nov 8 03:11:59 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=192.168.160.133 --slave_ip=192.168.160.133 --slave_port=3388 --workdir=/var/tmp --target_version=5.6.26-log --manager_version=0.57 --relay_log_info=/data/mysqldata/relay-log.info --relay_dir=/data/mysqldata/ --slave_pass=xxx
Thu Nov 8 03:11:59 2018 - [info] Connecting to [email protected](192.168.160.133:22).. 
Can‘t exec "mysqlbinlog": No such file or directory at /usr/share/perl5/vendor_perl/MHA/BinlogManager.pm line 106. 
mysqlbinlog version command failed with rc 1:0, please verify PATH, LD_LIBRARY_PATH, and client options #看报错信息，是说133这台服务器不能直接执行mysqlbinlog命令，解决方案，133服务器配置环境变量vim .bashrc   添加mysql路径
 at /usr/bin/apply_diff_relay_logs line 493
Thu Nov 8 03:12:00 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln208] Slaves settings check failed!
Thu Nov 8 03:12:00 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln416] Slave configuration failed.
Thu Nov 8 03:12:00 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/bin/masterha_check_repl line 48
Thu Nov 8 03:12:00 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.
Thu Nov 8 03:12:00 2018 - [info] Got exit code 1 (Not master dead).

MySQL Replication Health is NOT OK!

Thu Nov 8 17:27:21 2018 - [info] Checking replication health on 192.168.160.133..
Thu Nov 8 17:27:21 2018 - [info] ok.
Thu Nov 8 17:27:21 2018 - [info] Checking replication health on 192.168.160.134..
Thu Nov 8 17:27:21 2018 - [info] ok.
Thu Nov 8 17:27:21 2018 - [info] Checking master_ip_failover_script status:
Thu Nov 8 17:27:21 2018 - [info] /etc/mha/scripts/master_ip_failover_new --command=status --ssh_user=root --orig_master_host=192.168.160.132 --orig_master_ip=192.168.160.132 --orig_master_port=3388 
Thu Nov 8 17:27:21 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. Can‘t exec "/etc/mha/scripts/master_ip_failover_new": No such file or directory at /usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm line 68.  ##脚本没有上传
Thu Nov 8 17:27:21 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.
Thu Nov 8 17:27:21 2018 - [info] Got exit code 1 (Not master dead).

MySQL Replication Health is NOT OK!
Thu Nov 8 17:27:21 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln229] Failed to get master_ip_failover_script status with return code 1:0.
Thu Nov 8 17:27:21 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations. at /usr/bin/masterha_check_repl line 48
Thu Nov 8 17:27:21 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.
Thu Nov 8 17:27:21 2018 - [info] Got exit code 1 (Not master dead).

MySQL Replication Health is NOT OK!

再次检测

[[email protected] opt]# masterha_check_repl --conf=/etc/mha/conf/app1.conf 
Thu Nov 8 17:36:54 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Nov 8 17:36:54 2018 - [info] Reading application default configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 17:36:54 2018 - [info] Reading server configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 17:36:54 2018 - [info] MHA::MasterMonitor version 0.57.
Thu Nov 8 17:36:54 2018 - [info] GTID failover mode = 0
Thu Nov 8 17:36:54 2018 - [info] Dead Servers:
Thu Nov 8 17:36:54 2018 - [info] Alive Servers:
Thu Nov 8 17:36:54 2018 - [info] 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 17:36:54 2018 - [info] 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 17:36:54 2018 - [info] 192.168.160.134(192.168.160.134:3306)
Thu Nov 8 17:36:54 2018 - [info] Alive Slaves:
Thu Nov 8 17:36:54 2018 - [info] 192.168.160.133(192.168.160.133:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 17:36:54 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 17:36:54 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 17:36:54 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 17:36:54 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 17:36:54 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 17:36:54 2018 - [info] Current Alive Master: 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 17:36:54 2018 - [info] Checking slave configurations..
Thu Nov 8 17:36:54 2018 - [info] read_only=1 is not set on slave 192.168.160.133(192.168.160.133:3388).
Thu Nov 8 17:36:54 2018 - [warning] relay_log_purge=0 is not set on slave 192.168.160.133(192.168.160.133:3388).
Thu Nov 8 17:36:54 2018 - [info] read_only=1 is not set on slave 192.168.160.134(192.168.160.134:3306).
Thu Nov 8 17:36:54 2018 - [warning] relay_log_purge=0 is not set on slave 192.168.160.134(192.168.160.134:3306).
Thu Nov 8 17:36:54 2018 - [info] Checking replication filtering settings..
Thu Nov 8 17:36:54 2018 - [info] binlog_do_db= , binlog_ignore_db= 
Thu Nov 8 17:36:54 2018 - [info] Replication filtering check ok.
Thu Nov 8 17:36:54 2018 - [info] GTID (with auto-pos) is not supported
Thu Nov 8 17:36:54 2018 - [info] Starting SSH connection tests..
Thu Nov 8 17:37:04 2018 - [info] All SSH connection tests passed successfully.
Thu Nov 8 17:37:04 2018 - [info] Checking MHA Node version..
Thu Nov 8 17:37:09 2018 - [info] Version check ok.
Thu Nov 8 17:37:09 2018 - [info] Checking SSH publickey authentication settings on the current master..
Thu Nov 8 17:37:14 2018 - [info] HealthCheck: SSH to 192.168.160.132 is reachable.
Thu Nov 8 17:37:14 2018 - [info] Master MHA Node version is 0.57.
Thu Nov 8 17:37:14 2018 - [info] Checking recovery script configurations on 192.168.160.132(192.168.160.132:3388)..
Thu Nov 8 17:37:14 2018 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/log/binlog --output_file=/var/tmp/save_binary_logs_test --manager_version=0.57 --start_file=bin.000004 
Thu Nov 8 17:37:14 2018 - [info] Connecting to [email protected](192.168.160.132:22).. 
  Creating /var/tmp if not exists.. ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /log/binlog, up to bin.000004
Thu Nov 8 17:37:15 2018 - [info] Binlog setting check done.
Thu Nov 8 17:37:15 2018 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Thu Nov 8 17:37:15 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=192.168.160.133 --slave_ip=192.168.160.133 --slave_port=3388 --workdir=/var/tmp --target_version=5.6.26-log --manager_version=0.57 --relay_log_info=/data/mysqldata/relay-log.info --relay_dir=/data/mysqldata/ --slave_pass=xxx
Thu Nov 8 17:37:15 2018 - [info] Connecting to [email protected](192.168.160.133:22).. 
  Checking slave recovery environment settings..
    Opening /data/mysqldata/relay-log.info ... ok.
    Relay log found at /data/mysqldata, up to mysql57-relay-bin.000002
    Temporary relay log file is /data/mysqldata/mysql57-relay-bin.000002
    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Thu Nov 8 17:37:15 2018 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user=‘root‘ --slave_host=192.168.160.134 --slave_ip=192.168.160.134 --slave_port=3306 --workdir=/var/tmp --target_version=5.6.26-log --manager_version=0.57 --relay_log_info=/data/mysql/mysqldata3306/mydata/relay-log.info --relay_dir=/data/mysql/mysqldata3306/mydata/ --slave_pass=xxx
Thu Nov 8 17:37:15 2018 - [info] Connecting to [email protected](192.168.160.134:22).. 
  Checking slave recovery environment settings..
    Opening /data/mysql/mysqldata3306/mydata/relay-log.info ... ok.
    Relay log found at /data/mysql/mysqldata3306/relaylog, up to mysql_relay_bin.000002
    Temporary relay log file is /data/mysql/mysqldata3306/relaylog/mysql_relay_bin.000002
    Testing mysql connection and privileges..Warning: Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Thu Nov 8 17:37:16 2018 - [info] Slaves settings check done.
Thu Nov 8 17:37:16 2018 - [info] 
192.168.160.132(192.168.160.132:3388) (current master)
 +--192.168.160.133(192.168.160.133:3388)
 +--192.168.160.134(192.168.160.134:3306)

Thu Nov 8 17:37:16 2018 - [info] Checking replication health on 192.168.160.133..
Thu Nov 8 17:37:16 2018 - [info] ok.
Thu Nov 8 17:37:16 2018 - [info] Checking replication health on 192.168.160.134..
Thu Nov 8 17:37:16 2018 - [info] ok.
Thu Nov 8 17:37:16 2018 - [info] Checking master_ip_failover_script status:
Thu Nov 8 17:37:16 2018 - [info] /etc/mha/scripts/master_ip_failover_new --command=status --ssh_user=root --orig_master_host=192.168.160.132 --orig_master_ip=192.168.160.132 --orig_master_port=3388 
Checki ng the Status of the script.. OK 
Thu Nov 8 17:37:16 2018 - [info] OK.
Thu Nov 8 17:37:16 2018 - [warning] shutdown_script is not defined.
Thu Nov 8 17:37:16 2018 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.

5.启动MHA监控

nohup masterha_manager --conf=/etc/mha/conf/app1.conf --remove_dead_master_conf --ignore_last_failover < /dev/null >/etc/mha/log/manager.log 2>&1 &

6.查看状态

[[email protected] opt]# masterha_check_status --conf=/etc/mha/conf/app1.conf 
app1 (pid:2704) is running(0:PING_OK), master:192.168.160.132    #说明监控主是ok的

7.手动故障切换 (让主库挂掉)

[[email protected] opt]# masterha_check_status --conf=/etc/mha/conf/app1.conf 
app1 master maybe down(20:PING_FAILING). master:192.168.160.132             ###停掉主库，再次检测，提示 主库挂掉
Check /etc/mha/log/manager.log for details.

[[email protected] opt]# masterha_master_switch --master_state=dead --conf=/etc/mha/conf/app1.conf --dead_master_host=192.168.160.132 --dead_master_port=3388 --new_master_host=192.168.160.133 --new_master_port=3388 --ignore_last_failover     ##手动切换，将160.133作为新的主
--dead_master_ip=<dead_master_ip> is not set. Using 192.168.160.132.
Thu Nov 8 19:02:13 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Nov 8 19:02:13 2018 - [info] Reading application default configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 19:02:13 2018 - [info] Reading server configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 19:02:13 2018 - [info] MHA::MasterFailover version 0.57.
Thu Nov 8 19:02:13 2018 - [info] Starting master failover.
Thu Nov 8 19:02:13 2018 - [info] 
Thu Nov 8 19:02:13 2018 - [info] * Phase 1: Configuration Check Phase..
Thu Nov 8 19:02:13 2018 - [info] 
Thu Nov 8 19:02:14 2018 - [info] HealthCheck: SSH to 192.168.160.132 is reachable.
Thu Nov 8 19:02:15 2018 - [info] Binlog server 192.168.160.132 is reachable.
Thu Nov 8 19:02:15 2018 - [info] HealthCheck: SSH to 192.168.160.133 is reachable.
Thu Nov 8 19:02:16 2018 - [info] Binlog server 192.168.160.133 is reachable.
Thu Nov 8 19:02:16 2018 - [info] HealthCheck: SSH to 192.168.160.134 is reachable.
Thu Nov 8 19:02:16 2018 - [info] Binlog server 192.168.160.134 is reachable.
Thu Nov 8 19:02:17 2018 - [info] GTID failover mode = 0
Thu Nov 8 19:02:17 2018 - [info] Dead Servers:
Thu Nov 8 19:02:17 2018 - [info] 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 19:02:17 2018 - [info] Checking master reachability via MySQL(double check)...
Thu Nov 8 19:02:17 2018 - [info] ok.
Thu Nov 8 19:02:17 2018 - [info] Alive Servers:
Thu Nov 8 19:02:17 2018 - [info] 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 19:02:17 2018 - [info] 192.168.160.134(192.168.160.134:3306)
Thu Nov 8 19:02:17 2018 - [info] Alive Slaves:
Thu Nov 8 19:02:17 2018 - [info] 192.168.160.133(192.168.160.133:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 19:02:17 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 19:02:17 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 19:02:17 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 19:02:17 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 19:02:17 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Master 192.168.160.132(192.168.160.132:3388) is dead. Proceed? (yes/NO): yes
Thu Nov 8 19:02:24 2018 - [info] Starting Non-GTID based failover.
Thu Nov 8 19:02:24 2018 - [info] 
Thu Nov 8 19:02:24 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Thu Nov 8 19:02:24 2018 - [info] 
Thu Nov 8 19:02:24 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
Thu Nov 8 19:02:24 2018 - [info] 
Thu Nov 8 19:02:29 2018 - [warning] HealthCheck: Got timeout on checking SSH connection to 192.168.160.132! at /usr/share/perl5/vendor_perl/MHA/HealthCheck.pm line 342, <STDIN> line 1.
Thu Nov 8 19:02:29 2018 - [info] Forcing shutdown so that applications never connect to the current master..
Thu Nov 8 19:02:29 2018 - [info] Executing master IP deactivation script:
Thu Nov 8 19:02:29 2018 - [info] /etc/mha/scripts/master_ip_failover_new --orig_master_host=192.168.160.132 --orig_master_ip=192.168.160.132 --orig_master_port=3388 --command=stop 
Disabling the VIP on old master: 192.168.160.132 
Thu Nov 8 19:02:29 2018 - [info] done.
Thu Nov 8 19:02:29 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Thu Nov 8 19:02:29 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Thu Nov 8 19:02:29 2018 - [info] 
Thu Nov 8 19:02:29 2018 - [info] * Phase 3: Master Recovery Phase..
Thu Nov 8 19:02:29 2018 - [info] 
Thu Nov 8 19:02:29 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Thu Nov 8 19:02:29 2018 - [info] 
Thu Nov 8 19:02:29 2018 - [info] The latest binary log file/position on all slaves is bin.000004:996
Thu Nov 8 19:02:29 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
Thu Nov 8 19:02:29 2018 - [info] 192.168.160.133(192.168.160.133:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 19:02:29 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 19:02:29 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 19:02:29 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 19:02:29 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 19:02:29 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 19:02:29 2018 - [info] The oldest binary log file/position on all slaves is bin.000004:996
Thu Nov 8 19:02:29 2018 - [info] Oldest slaves:
Thu Nov 8 19:02:29 2018 - [info] 192.168.160.133(192.168.160.133:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 19:02:29 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 19:02:29 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 19:02:29 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 19:02:29 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 19:02:29 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 19:02:29 2018 - [info] 
Thu Nov 8 19:02:29 2018 - [info] * Phase 3.2: Saving Dead Master‘s Binlog Phase..
Thu Nov 8 19:02:29 2018 - [info] 
Thu Nov 8 19:02:29 2018 - [warning] Dead Master is not SSH reachable. Could not save it‘s binlogs. Transactions that were not sent to the latest slave (Read_Master_Log_Pos to the tail of the dead master‘s binlog) were lost.
Thu Nov 8 19:02:29 2018 - [info] 
Thu Nov 8 19:02:29 2018 - [info] * Phase 3.3: Determining New Master Phase..
Thu Nov 8 19:02:29 2018 - [info] 
Thu Nov 8 19:02:29 2018 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Thu Nov 8 19:02:29 2018 - [info] All slaves received relay logs to the same position. No need to resync each other.
Thu Nov 8 19:02:29 2018 - [info] 192.168.160.133 can be new master.
Thu Nov 8 19:02:29 2018 - [info] New master is 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 19:02:29 2018 - [info] Starting master failover..
Thu Nov 8 19:02:29 2018 - [info] 
From:
192.168.160.132(192.168.160.132:3388) (current master)
 +--192.168.160.133(192.168.160.133:3388)
 +--192.168.160.134(192.168.160.134:3306)

To:
192.168.160.133(192.168.160.133:3388) (new master)
 +--192.168.160.134(192.168.160.134:3306)

Starting master switch from 192.168.160.132(192.168.160.132:3388) to 192.168.160.133(192.168.160.133:3388)? (yes/NO): yes
Thu Nov 8 19:02:35 2018 - [info] New master decided manually is 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] * Phase 3.4: Master Log Apply Phase..
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Thu Nov 8 19:02:35 2018 - [info] Starting recovery on 192.168.160.133(192.168.160.133:3388)..
Thu Nov 8 19:02:35 2018 - [info] This server has all relay logs. Waiting all logs to be applied.. 
Thu Nov 8 19:02:35 2018 - [info] done.
Thu Nov 8 19:02:35 2018 - [info] All relay logs were successfully applied.
Thu Nov 8 19:02:35 2018 - [info] Getting new master‘s binlog name and position..
Thu Nov 8 19:02:35 2018 - [info] bin.000006:996
Thu Nov 8 19:02:35 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST=‘192.168.160.133‘, MASTER_PORT=3388, MASTER_LOG_FILE=‘bin.000006‘, MASTER_LOG_POS=996, MASTER_USER=‘replication_user‘, MASTER_PASSWORD=‘xxx‘;
Thu Nov 8 19:02:35 2018 - [info] Executing master IP activate script:
Thu Nov 8 19:02:35 2018 - [info] /etc/mha/scripts/master_ip_failover_new --command=start --ssh_user=root --orig_master_host=192.168.160.132 --orig_master_ip=192.168.160.132 --orig_master_port=3388 --new_master_host=192.168.160.133 --new_master_ip=192.168.160.133 --new_master_port=3388 --new_master_user=‘root‘ --new_master_password=xxx
Enabling the VIP - 192.168.200.15/24 on the new master - 192.168.160.133 
Use of uninitialized value $new_if in concatenation (.) or string at /etc/mha/scripts/master_ip_failover_new line 98.
Thu Nov 8 19:02:35 2018 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln1589] Failed to activate master IP address for 192.168.160.133(192.168.160.133:3388) with return code 10:0
Thu Nov 8 19:02:35 2018 - [warning] Proceeding.
Thu Nov 8 19:02:35 2018 - [info] ** Finished master recovery successfully.
Thu Nov 8 19:02:35 2018 - [info] * Phase 3: Master Recovery Phase completed.
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] * Phase 4: Slaves Recovery Phase..
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] -- Slave diff file generation on host 192.168.160.134(192.168.160.134:3306) started, pid: 3079. Check tmp log /etc/mha/app1/192.168.160.134_3306_20181108190213.log if it takes time..
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] Log messages from 192.168.160.134 ...
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Thu Nov 8 19:02:35 2018 - [info] End of log messages from 192.168.160.134.
Thu Nov 8 19:02:35 2018 - [info] -- 192.168.160.134(192.168.160.134:3306) has the latest relay log events.
Thu Nov 8 19:02:35 2018 - [info] Generating relay diff files from the latest slave succeeded.
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] -- Slave recovery on host 192.168.160.134(192.168.160.134:3306) started, pid: 3081. Check tmp log /etc/mha/app1/192.168.160.134_3306_20181108190213.log if it takes time..
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] Log messages from 192.168.160.134 ...
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] Starting recovery on 192.168.160.134(192.168.160.134:3306)..
Thu Nov 8 19:02:35 2018 - [info] This server has all relay logs. Waiting all logs to be applied.. 
Thu Nov 8 19:02:35 2018 - [info] done.
Thu Nov 8 19:02:35 2018 - [info] All relay logs were successfully applied.
Thu Nov 8 19:02:35 2018 - [info] Resetting slave 192.168.160.134(192.168.160.134:3306) and starting replication from the new master 192.168.160.133(192.168.160.133:3388)..
Thu Nov 8 19:02:35 2018 - [info] Executed CHANGE MASTER.
Thu Nov 8 19:02:35 2018 - [info] Slave started.
Thu Nov 8 19:02:35 2018 - [info] End of log messages from 192.168.160.134.
Thu Nov 8 19:02:35 2018 - [info] -- Slave recovery on host 192.168.160.134(192.168.160.134:3306) succeeded.
Thu Nov 8 19:02:35 2018 - [info] All new slave servers recovered successfully.
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] * Phase 5: New master cleanup phase..
Thu Nov 8 19:02:35 2018 - [info] 
Thu Nov 8 19:02:35 2018 - [info] Resetting slave info on the new master..
Thu Nov 8 19:02:35 2018 - [info] 192.168.160.133: Resetting slave info succeeded.
Thu Nov 8 19:02:35 2018 - [info] Master failover to 192.168.160.133(192.168.160.133:3388) completed successfully.
Thu Nov 8 19:02:35 2018 - [info] 

----- Failover Report -----

app1: MySQL Master failover 192.168.160.132(192.168.160.132:3388) to 192.168.160.133(192.168.160.133:3388) succeeded

Master 192.168.160.132(192.168.160.132:3388) is down!

Check MHA Manager logs at localhost.localdomain for details.

Started manual(interactive) failover.
Invalidated master IP address on 192.168.160.132(192.168.160.132:3388)
The latest slave 192.168.160.133(192.168.160.133:3388) has all relay logs for recovery.
Selected 192.168.160.133(192.168.160.133:3388) as a new master.
192.168.160.133(192.168.160.133:3388): OK: Applying all logs succeeded.
Failed to activate master IP address for 192.168.160.133(192.168.160.133:3388) with return code 10:0
192.168.160.134(192.168.160.134:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.160.134(192.168.160.134:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.160.133(192.168.160.133:3388)
192.168.160.133(192.168.160.133:3388): Resetting slave info succeeded.
Master failover to 192.168.160.133(192.168.160.133:3388) completed successfully.   #切换成功，在手动切换的时候，会提示是否要切换

8.手工正常切换

[[email protected] opt]# masterha_master_switch --conf=/etc/mha/conf/app1.conf --master_state=alive --new_master_host=192.168.160.132 --new_master_port=3388 --orig_master_is_new_slave --running_updates_limit=10000
Thu Nov 8 22:26:56 2018 - [info] MHA::MasterRotate version 0.57.
Thu Nov 8 22:26:56 2018 - [info] Starting online master switch..
Thu Nov 8 22:26:56 2018 - [info] 
Thu Nov 8 22:26:56 2018 - [info] * Phase 1: Configuration Check Phase..
Thu Nov 8 22:26:56 2018 - [info] 
Thu Nov 8 22:26:56 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Nov 8 22:26:56 2018 - [info] Reading application default configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 22:26:56 2018 - [info] Reading server configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 22:26:56 2018 - [info] GTID failover mode = 0
Thu Nov 8 22:26:56 2018 - [info] Current Alive Master: 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 22:26:56 2018 - [info] Alive Slaves:
Thu Nov 8 22:26:56 2018 - [info] 192.168.160.132(192.168.160.132:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 22:26:56 2018 - [info] Replicating from 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 22:26:56 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 22:26:56 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 22:26:56 2018 - [info] Replicating from 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 22:26:56 2018 - [info] Primary candidate for the new Master (candidate_master is set)

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.160.133(192.168.160.133:3388)? (YES/no): yes
Thu Nov 8 22:27:06 2018 - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Thu Nov 8 22:27:06 2018 - [info] ok.
Thu Nov 8 22:27:06 2018 - [info] Checking MHA is not monitoring or doing failover..
Thu Nov 8 22:27:06 2018 - [info] Checking replication health on 192.168.160.132..
Thu Nov 8 22:27:06 2018 - [info] ok.
Thu Nov 8 22:27:06 2018 - [info] Checking replication health on 192.168.160.134..
Thu Nov 8 22:27:06 2018 - [info] ok.
Thu Nov 8 22:27:06 2018 - [info] 192.168.160.132 can be new master.
Thu Nov 8 22:27:06 2018 - [info] 
From:
192.168.160.133(192.168.160.133:3388) (current master)
 +--192.168.160.132(192.168.160.132:3388)
 +--192.168.160.134(192.168.160.134:3306)

To:
192.168.160.132(192.168.160.132:3388) (new master)
 +--192.168.160.134(192.168.160.134:3306)
 +--192.168.160.133(192.168.160.133:3388)

Starting master switch from 192.168.160.133(192.168.160.133:3388) to 192.168.160.132(192.168.160.132:3388)? (yes/NO): yes
Thu Nov 8 22:27:15 2018 - [info] Checking whether 192.168.160.132(192.168.160.132:3388) is ok for the new master..
Thu Nov 8 22:27:15 2018 - [info] ok.
Thu Nov 8 22:27:15 2018 - [info] 192.168.160.133(192.168.160.133:3388): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Thu Nov 8 22:27:15 2018 - [info] 192.168.160.133(192.168.160.133:3388): Resetting slave pointing to the dummy host.
Thu Nov 8 22:27:15 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Thu Nov 8 22:27:15 2018 - [info] 
Thu Nov 8 22:27:15 2018 - [info] * Phase 2: Rejecting updates Phase..
Thu Nov 8 22:27:15 2018 - [info] 
Thu Nov 8 22:27:15 2018 - [info] Executing master ip online change script to disable write on the current master:
Thu Nov 8 22:27:15 2018 - [info] /etc/mha/scripts/master_online_change_new --command=stop --orig_master_host=192.168.160.133 --orig_master_ip=192.168.160.133 --orig_master_port=3388 --orig_master_user=‘root‘ --new_master_host=192.168.160.132 --new_master_ip=192.168.160.132 --new_master_port=3388 --new_master_user=‘root‘ --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Thu Nov 8 22:27:15 2018 939648 Set read_only on the new master.. ok.
Thu Nov 8 22:27:15 2018 943106 drop vip 192.168.160.15/24..
Thu Nov 8 22:27:16 2018 265056 Set read_only=1 on the orig master.. ok.
Thu Nov 8 22:27:16 2018 267916 Killing all application threads..
Thu Nov 8 22:27:16 2018 267955 done.
Thu Nov 8 22:27:16 2018 - [info] ok.
Thu Nov 8 22:27:16 2018 - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Thu Nov 8 22:27:16 2018 - [info] Executing FLUSH TABLES WITH READ LOCK..
Thu Nov 8 22:27:16 2018 - [info] ok.
Thu Nov 8 22:27:16 2018 - [info] Orig master binlog:pos is bin.000007:208.
Thu Nov 8 22:27:16 2018 - [info] Waiting to execute all relay logs on 192.168.160.132(192.168.160.132:3388)..
Thu Nov 8 22:27:16 2018 - [info] master_pos_wait(bin.000007:208) completed on 192.168.160.132(192.168.160.132:3388). Executed 0 events.
Thu Nov 8 22:27:16 2018 - [info] done.
Thu Nov 8 22:27:16 2018 - [info] Getting new master‘s binlog name and position..
Thu Nov 8 22:27:16 2018 - [info] bin.000007:120
Thu Nov 8 22:27:16 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST=‘192.168.160.132‘, MASTER_PORT=3388, MASTER_LOG_FILE=‘bin.000007‘, MASTER_LOG_POS=120, MASTER_USER=‘replication_user‘, MASTER_PASSWORD=‘xxx‘;   ###切换日志中获取主信息
Thu Nov 8 22:27:16 2018 - [info] Executing master ip online change script to allow write on the new master:
Thu Nov 8 22:27:16 2018 - [info] /etc/mha/scripts/master_online_change_new --command=start --orig_master_host=192.168.160.133 --orig_master_ip=192.168.160.133 --orig_master_port=3388 --orig_master_user=‘root‘ --new_master_host=192.168.160.132 --new_master_ip=192.168.160.132 --new_master_port=3388 --new_master_user=‘root‘ --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_is_new_slave --orig_master_password=xxx --new_master_password=xxx
Thu Nov 8 22:27:16 2018 339916 Set read_only=0 on the new master.
Thu Nov 8 22:27:16 2018 340763Add vip 192.168.160.15/24 on eth1..
Thu Nov 8 22:27:16 2018 - [info] ok.
Thu Nov 8 22:27:16 2018 - [info] 
Thu Nov 8 22:27:16 2018 - [info] * Switching slaves in parallel..
Thu Nov 8 22:27:16 2018 - [info] 
Thu Nov 8 22:27:16 2018 - [info] -- Slave switch on host 192.168.160.134(192.168.160.134:3306) started, pid: 3409
Thu Nov 8 22:27:16 2018 - [info] 
Thu Nov 8 22:27:16 2018 - [info] Log messages from 192.168.160.134 ...
Thu Nov 8 22:27:16 2018 - [info] 
Thu Nov 8 22:27:16 2018 - [info] Waiting to execute all relay logs on 192.168.160.134(192.168.160.134:3306)..
Thu Nov 8 22:27:16 2018 - [info] master_pos_wait(bin.000007:208) completed on 192.168.160.134(192.168.160.134:3306). Executed 0 events.
Thu Nov 8 22:27:16 2018 - [info] done.
Thu Nov 8 22:27:16 2018 - [info] Resetting slave 192.168.160.134(192.168.160.134:3306) and starting replication from the new master 192.168.160.132(192.168.160.132:3388)..
Thu Nov 8 22:27:16 2018 - [info] Executed CHANGE MASTER.
Thu Nov 8 22:27:16 2018 - [info] Slave started.
Thu Nov 8 22:27:16 2018 - [info] End of log messages from 192.168.160.134 ...
Thu Nov 8 22:27:16 2018 - [info] 
Thu Nov 8 22:27:16 2018 - [info] -- Slave switch on host 192.168.160.134(192.168.160.134:3306) succeeded.
Thu Nov 8 22:27:16 2018 - [info] Unlocking all tables on the orig master:
Thu Nov 8 22:27:16 2018 - [info] Executing UNLOCK TABLES..
Thu Nov 8 22:27:16 2018 - [info] ok.
Thu Nov 8 22:27:16 2018 - [info] Starting orig master as a new slave..
Thu Nov 8 22:27:16 2018 - [info] Resetting slave 192.168.160.133(192.168.160.133:3388) and starting replication from the new master 192.168.160.132(192.168.160.132:3388)..
Thu Nov 8 22:27:16 2018 - [info] Executed CHANGE MASTER.
Thu Nov 8 22:27:16 2018 - [info] Slave started.
Thu Nov 8 22:27:16 2018 - [info] All new slave servers switched successfully.
Thu Nov 8 22:27:16 2018 - [info] 
Thu Nov 8 22:27:16 2018 - [info] * Phase 5: New master cleanup phase..
Thu Nov 8 22:27:16 2018 - [info] 
Thu Nov 8 22:27:16 2018 - [info] 192.168.160.132: Resetting slave info succeeded.
Thu Nov 8 22:27:16 2018 - [info] Switching master to 192.168.160.132(192.168.160.132:3388) completed successfully.

####从库设置为read_only测试
1.将从库设置为read_only


mysql> show variables like ‘%read_only%‘;
+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| innodb_read_only | OFF |
| read_only | ON |
| tx_read_only | OFF |
+------------------+-------+
3 rows in set (0.00 sec)

2.手动故障切换

[[email protected] opt]# masterha_master_switch --master_state=dead --conf=/etc/mha/conf/app1.conf --dead_master_host=192.168.160.132 --dead_master_port=3388 --new_master_host=192.168.160.133 --new_master_port=3388 --ignore_last_failover
--dead_master_ip=<dead_master_ip> is not set. Using 192.168.160.132.
Thu Nov 8 23:56:02 2018 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Thu Nov 8 23:56:02 2018 - [info] Reading application default configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 23:56:02 2018 - [info] Reading server configuration from /etc/mha/conf/app1.conf..
Thu Nov 8 23:56:02 2018 - [info] MHA::MasterFailover version 0.57.
Thu Nov 8 23:56:02 2018 - [info] Starting master failover.
Thu Nov 8 23:56:02 2018 - [info] 
Thu Nov 8 23:56:02 2018 - [info] * Phase 1: Configuration Check Phase..
Thu Nov 8 23:56:02 2018 - [info] 
Thu Nov 8 23:56:02 2018 - [info] HealthCheck: SSH to 192.168.160.132 is reachable.
Thu Nov 8 23:56:02 2018 - [info] Binlog server 192.168.160.132 is reachable.
Thu Nov 8 23:56:03 2018 - [info] HealthCheck: SSH to 192.168.160.133 is reachable.
Thu Nov 8 23:56:03 2018 - [info] Binlog server 192.168.160.133 is reachable.
Thu Nov 8 23:56:03 2018 - [info] HealthCheck: SSH to 192.168.160.134 is reachable.
Thu Nov 8 23:56:04 2018 - [info] Binlog server 192.168.160.134 is reachable.
Thu Nov 8 23:56:04 2018 - [info] GTID failover mode = 0
Thu Nov 8 23:56:04 2018 - [info] Dead Servers:
Thu Nov 8 23:56:04 2018 - [info] 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 23:56:04 2018 - [info] Checking master reachability via MySQL(double check)...
Thu Nov 8 23:56:04 2018 - [info] ok.
Thu Nov 8 23:56:04 2018 - [info] Alive Servers:
Thu Nov 8 23:56:04 2018 - [info] 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 23:56:04 2018 - [info] 192.168.160.134(192.168.160.134:3306)
Thu Nov 8 23:56:04 2018 - [info] Alive Slaves:
Thu Nov 8 23:56:04 2018 - [info] 192.168.160.133(192.168.160.133:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 23:56:04 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 23:56:04 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 23:56:04 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 23:56:04 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 23:56:04 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Master 192.168.160.132(192.168.160.132:3388) is dead. Proceed? (yes/NO): yes
Thu Nov 8 23:56:05 2018 - [info] Starting Non-GTID based failover.
Thu Nov 8 23:56:05 2018 - [info] 
Thu Nov 8 23:56:05 2018 - [info] ** Phase 1: Configuration Check Phase completed.
Thu Nov 8 23:56:05 2018 - [info] 
Thu Nov 8 23:56:05 2018 - [info] * Phase 2: Dead Master Shutdown Phase..
Thu Nov 8 23:56:05 2018 - [info] 
Thu Nov 8 23:56:09 2018 - [info] HealthCheck: SSH to 192.168.160.132 is reachable.
Thu Nov 8 23:56:10 2018 - [info] Forcing shutdown so that applications never connect to the current master..
Thu Nov 8 23:56:10 2018 - [info] Executing master IP deactivation script:
Thu Nov 8 23:56:10 2018 - [info] /etc/mha/scripts/master_ip_failover_new --orig_master_host=192.168.160.132 --orig_master_ip=192.168.160.132 --orig_master_port=3388 --command=stopssh --ssh_user=root  
Disabling the VIP on old master: 192.168.160.132 
RTNETLINK answers: Cannot assign requested address
Thu Nov 8 23:56:14 2018 - [info] done.
Thu Nov 8 23:56:14 2018 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Thu Nov 8 23:56:14 2018 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Thu Nov 8 23:56:14 2018 - [info] 
Thu Nov 8 23:56:14 2018 - [info] * Phase 3: Master Recovery Phase..
Thu Nov 8 23:56:14 2018 - [info] 
Thu Nov 8 23:56:14 2018 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Thu Nov 8 23:56:14 2018 - [info] 
Thu Nov 8 23:56:14 2018 - [info] The latest binary log file/position on all slaves is bin.000011:120
Thu Nov 8 23:56:14 2018 - [info] Latest slaves (Slaves that received relay log files to the latest):
Thu Nov 8 23:56:14 2018 - [info] 192.168.160.133(192.168.160.133:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 23:56:14 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 23:56:14 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 23:56:14 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 23:56:14 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 23:56:14 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 23:56:14 2018 - [info] The oldest binary log file/position on all slaves is bin.000011:120
Thu Nov 8 23:56:14 2018 - [info] Oldest slaves:
Thu Nov 8 23:56:14 2018 - [info] 192.168.160.133(192.168.160.133:3388) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 23:56:14 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 23:56:14 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 23:56:14 2018 - [info] 192.168.160.134(192.168.160.134:3306) Version=5.6.26-log (oldest major version between slaves) log-bin:enabled
Thu Nov 8 23:56:14 2018 - [info] Replicating from 192.168.160.132(192.168.160.132:3388)
Thu Nov 8 23:56:14 2018 - [info] Primary candidate for the new Master (candidate_master is set)
Thu Nov 8 23:56:14 2018 - [info] 
Thu Nov 8 23:56:14 2018 - [info] * Phase 3.2: Saving Dead Master‘s Binlog Phase..
Thu Nov 8 23:56:14 2018 - [info] 
Thu Nov 8 23:56:19 2018 - [info] Fetching dead master‘s binary logs..
Thu Nov 8 23:56:19 2018 - [info] Executing command on the dead master 192.168.160.132(192.168.160.132:3388): save_binary_logs --command=save --start_file=bin.000011 --start_pos=120 --binlog_dir=/log/binlog --output_file=/var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57
  Creating /var/tmp if not exists.. ok.
 Concat binary/relay logs from bin.000011 pos 120 to bin.000011 EOF into /var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog ..
 Binlog Checksum enabled
  Dumping binlog format description event, from position 0 to 120.. ok.
  Dumping effective binlog data from /log/binlog/bin.000011 position 120 to tail(143).. ok.
 Binlog Checksum enabled
 Concat succeeded.
saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog 100% 143 0.1KB/s 00:00    
Thu Nov 8 23:56:24 2018 - [info] scp from [email protected]:/var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog to local:/etc/mha/app1/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog succeeded.
Thu Nov 8 23:56:24 2018 - [info] HealthCheck: SSH to 192.168.160.133 is reachable.
Thu Nov 8 23:56:25 2018 - [info] HealthCheck: SSH to 192.168.160.134 is reachable.
Thu Nov 8 23:56:26 2018 - [info] 
Thu Nov 8 23:56:26 2018 - [info] * Phase 3.3: Determining New Master Phase..
Thu Nov 8 23:56:26 2018 - [info] 
Thu Nov 8 23:56:26 2018 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Thu Nov 8 23:56:26 2018 - [info] All slaves received relay logs to the same position. No need to resync each other.
Thu Nov 8 23:56:26 2018 - [info] 192.168.160.133 can be new master.
Thu Nov 8 23:56:26 2018 - [info] New master is 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 23:56:26 2018 - [info] Starting master failover..
Thu Nov 8 23:56:26 2018 - [info] 
From:
192.168.160.132(192.168.160.132:3388) (current master)
 +--192.168.160.133(192.168.160.133:3388)
 +--192.168.160.134(192.168.160.134:3306)

To:
192.168.160.133(192.168.160.133:3388) (new master)
 +--192.168.160.134(192.168.160.134:3306)

Starting master switch from 192.168.160.132(192.168.160.132:3388) to 192.168.160.133(192.168.160.133:3388)? (yes/NO): yes
Thu Nov 8 23:56:31 2018 - [info] New master decided manually is 192.168.160.133(192.168.160.133:3388)
Thu Nov 8 23:56:31 2018 - [info] 
Thu Nov 8 23:56:31 2018 - [info] * Phase 3.3: New Master Diff Log Generation Phase..
Thu Nov 8 23:56:31 2018 - [info] 
Thu Nov 8 23:56:31 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Thu Nov 8 23:56:31 2018 - [info] Sending binlog..
saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog 100% 143 0.1KB/s 00:00    
Thu Nov 8 23:56:36 2018 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog to [email protected]:/var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog succeeded.
Thu Nov 8 23:56:36 2018 - [info] 
Thu Nov 8 23:56:36 2018 - [info] * Phase 3.4: Master Log Apply Phase..
Thu Nov 8 23:56:36 2018 - [info] 
Thu Nov 8 23:56:36 2018 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Thu Nov 8 23:56:36 2018 - [info] Starting recovery on 192.168.160.133(192.168.160.133:3388)..
Thu Nov 8 23:56:36 2018 - [info] Generating diffs succeeded.
Thu Nov 8 23:56:36 2018 - [info] Waiting until all relay logs are applied.
Thu Nov 8 23:56:36 2018 - [info] done.
Thu Nov 8 23:56:36 2018 - [info] Getting slave status..
Thu Nov 8 23:56:36 2018 - [info] This slave(192.168.160.133)‘s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(bin.000011:120). No need to recover from Exec_Master_Log_Pos.
Thu Nov 8 23:56:36 2018 - [info] Connecting to the target slave host 192.168.160.133, running recover script..
Thu Nov 8 23:56:36 2018 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user=‘root‘ --slave_host=192.168.160.133 --slave_ip=192.168.160.133 --slave_port=3388 --apply_files=/var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog --workdir=/var/tmp --target_version=5.6.26-log --timestamp=20181108235602 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --slave_pass=xxx
Thu Nov 8 23:56:36 2018 - [info] 
MySQL client version is 5.6.26. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog on 192.168.160.133:3388. This may take long time...
Applying log files succeeded.
Thu Nov 8 23:56:36 2018 - [info] All relay logs were successfully applied.
Thu Nov 8 23:56:36 2018 - [info] Getting new master‘s binlog name and position..
Thu Nov 8 23:56:36 2018 - [info] bin.000002:120
Thu Nov 8 23:56:36 2018 - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST=‘192.168.160.133‘, MASTER_PORT=3388, MASTER_LOG_FILE=‘bin.000002‘, MASTER_LOG_POS=120, MASTER_USER=‘replication_user‘, MASTER_PASSWORD=‘xxx‘;
Thu Nov 8 23:56:36 2018 - [info] Executing master IP activate script:
Thu Nov 8 23:56:36 2018 - [info] /etc/mha/scripts/master_ip_failover_new --command=start --ssh_user=root --orig_master_host=192.168.160.132 --orig_master_ip=192.168.160.132 --orig_master_port=3388 --new_master_host=192.168.160.133 --new_master_ip=192.168.160.133 --new_master_port=3388 --new_master_user=‘root‘ --new_master_password=xxx
Enabling the VIP - 192.168.160.15/24 on the new master - 192.168.160.133 
RTNETLINK answers: File exists
Thu Nov 8 23:56:37 2018 - [info] OK.
Thu Nov 8 23:56:37 2018 - [info] Setting read_only=0 on 192.168.160.133(192.168.160.133:3388)..
Thu Nov 8 23:56:37 2018 - [info] ok.
Thu Nov 8 23:56:37 2018 - [info] ** Finished master recovery successfully.
Thu Nov 8 23:56:37 2018 - [info] * Phase 3: Master Recovery Phase completed.
Thu Nov 8 23:56:37 2018 - [info] 
Thu Nov 8 23:56:37 2018 - [info] * Phase 4: Slaves Recovery Phase..
Thu Nov 8 23:56:37 2018 - [info] 
Thu Nov 8 23:56:37 2018 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Thu Nov 8 23:56:37 2018 - [info] 
Thu Nov 8 23:56:37 2018 - [info] -- Slave diff file generation on host 192.168.160.134(192.168.160.134:3306) started, pid: 3947. Check tmp log /etc/mha/app1/192.168.160.134_3306_20181108235602.log if it takes time..
Thu Nov 8 23:56:37 2018 - [info] 
Thu Nov 8 23:56:37 2018 - [info] Log messages from 192.168.160.134 ...
Thu Nov 8 23:56:37 2018 - [info] 
Thu Nov 8 23:56:37 2018 - [info] This server has all relay logs. No need to generate diff files from the latest slave.
Thu Nov 8 23:56:37 2018 - [info] End of log messages from 192.168.160.134.
Thu Nov 8 23:56:37 2018 - [info] -- 192.168.160.134(192.168.160.134:3306) has the latest relay log events.
Thu Nov 8 23:56:37 2018 - [info] Generating relay diff files from the latest slave succeeded.
Thu Nov 8 23:56:37 2018 - [info] 
Thu Nov 8 23:56:37 2018 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Thu Nov 8 23:56:37 2018 - [info] 
Thu Nov 8 23:56:37 2018 - [info] -- Slave recovery on host 192.168.160.134(192.168.160.134:3306) started, pid: 3949. Check tmp log /etc/mha/app1/192.168.160.134_3306_20181108235602.log if it takes time..
saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog 100% 143 0.1KB/s 00:00    
Thu Nov 8 23:56:38 2018 - [info] 
Thu Nov 8 23:56:38 2018 - [info] Log messages from 192.168.160.134 ...
Thu Nov 8 23:56:38 2018 - [info] 
Thu Nov 8 23:56:37 2018 - [info] Sending binlog..
Thu Nov 8 23:56:37 2018 - [info] scp from local:/etc/mha/app1/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog to [email protected]:/var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog succeeded.
Thu Nov 8 23:56:37 2018 - [info] Starting recovery on 192.168.160.134(192.168.160.134:3306)..
Thu Nov 8 23:56:37 2018 - [info] Generating diffs succeeded.
Thu Nov 8 23:56:37 2018 - [info] Waiting until all relay logs are applied.
Thu Nov 8 23:56:37 2018 - [info] done.
Thu Nov 8 23:56:37 2018 - [info] Getting slave status..
Thu Nov 8 23:56:37 2018 - [info] This slave(192.168.160.134)‘s Exec_Master_Log_Pos equals to Read_Master_Log_Pos(bin.000011:120). No need to recover from Exec_Master_Log_Pos.
Thu Nov 8 23:56:37 2018 - [info] Connecting to the target slave host 192.168.160.134, running recover script..
Thu Nov 8 23:56:37 2018 - [info] Executing command: apply_diff_relay_logs --command=apply --slave_user=‘root‘ --slave_host=192.168.160.134 --slave_ip=192.168.160.134 --slave_port=3306 --apply_files=/var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog --workdir=/var/tmp --target_version=5.6.26-log --timestamp=20181108235602 --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.57 --slave_pass=xxx
Thu Nov 8 23:56:38 2018 - [info] 
MySQL client version is 5.6.26. Using --binary-mode.
Applying differential binary/relay log files /var/tmp/saved_master_binlog_from_192.168.160.132_3388_20181108235602.binlog on 192.168.160.134:3306. This may take long time...
Applying log files succeeded.
Thu Nov 8 23:56:38 2018 - [info] All relay logs were successfully applied.
Thu Nov 8 23:56:38 2018 - [info] Resetting slave 192.168.160.134(192.168.160.134:3306) and starting replication from the new master 192.168.160.133(192.168.160.133:3388)..
Thu Nov 8 23:56:38 2018 - [info] Executed CHANGE MASTER.
Thu Nov 8 23:56:38 2018 - [info] Slave started.
Thu Nov 8 23:56:38 2018 - [info] End of log messages from 192.168.160.134.
Thu Nov 8 23:56:38 2018 - [info] -- Slave recovery on host 192.168.160.134(192.168.160.134:3306) succeeded.
Thu Nov 8 23:56:38 2018 - [info] All new slave servers recovered successfully.
Thu Nov 8 23:56:38 2018 - [info] 
Thu Nov 8 23:56:38 2018 - [info] * Phase 5: New master cleanup phase..
Thu Nov 8 23:56:38 2018 - [info] 
Thu Nov 8 23:56:38 2018 - [info] Resetting slave info on the new master..
Thu Nov 8 23:56:38 2018 - [info] 192.168.160.133: Resetting slave info succeeded.
Thu Nov 8 23:56:38 2018 - [info] Master failover to 192.168.160.133(192.168.160.133:3388) completed successfully.
Thu Nov 8 23:56:38 2018 - [info] 

----- Failover Report -----

app1: MySQL Master failover 192.168.160.132(192.168.160.132:3388) to 192.168.160.133(192.168.160.133:3388) succeeded

Master 192.168.160.132(192.168.160.132:3388) is down!

Check MHA Manager logs at localhost.localdomain for details.

Started manual(interactive) failover.
Invalidated master IP address on 192.168.160.132(192.168.160.132:3388)
The latest slave 192.168.160.133(192.168.160.133:3388) has all relay logs for recovery.
Selected 192.168.160.133(192.168.160.133:3388) as a new master.
192.168.160.133(192.168.160.133:3388): OK: Applying all logs succeeded.
192.168.160.133(192.168.160.133:3388): OK: Activated master IP address.
192.168.160.134(192.168.160.134:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.160.134(192.168.160.134:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.160.133(192.168.160.133:3388)
192.168.160.133(192.168.160.133:3388): Resetting slave info succeeded.
Master failover to 192.168.160.133(192.168.160.133:3388) completed successfully.

3.查看新主的read_only

mysql> show variables like ‘%read_only‘;
+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| innodb_read_only | OFF |
| read_only | OFF |
| tx_read_only | OFF |
+------------------+-------+
3 rows in set (0.00 sec)

4.将旧的主加入到新的集群，作为从库

mysql> show variables like ‘read_only‘;  ##旧的master
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only | OFF |
+---------------+-------+
1 row in set (0.00 sec)
mysql> set global read_only=1; ##将旧主设置类read_only
Query OK, 0 rows affected (0.00 sec)

mysql> 
mysql> show variables like ‘read_only‘;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only | ON |
+---------------+-------+
1 row in set (0.00 sec)

以上是关于MHA 安装及测试的主要内容，如果未能解决你的问题，请参考以下文章

Linux——MySQL MHA高可用配置及故障切换以及修复

MHA你难道不了解一下？mysql数据库MHA高可用配置及故障切换！

第3天一篇MHA安装（单实例）

MySQL高可用架构之MHA（理论+部署+故障模拟）

MHA高可用及读写分离

MySQL------MHA高可用配置及故障切换