keepalived裂脑导致主备无法切换 #yyds干货盘点#

Posted 2022-04-02 江晓龙的技术博客

1.keepalived高可用裂脑

由于某些原因，导致两台keepalived高可用服务器在指定时间内，无法检测到对方的心跳消息，各自取得资源及服务的所有权，而此时的两台高可用服务器又都还或者。

导致裂脑的原因：

​ 1.服务器网线松动等网络故障

​ 2.服务器硬件故障发生损坏现象而崩溃

​ 3.主备都开启firewalld防火墙

​ 4.nginx服务器死掉

2.模拟裂脑案例1-防火墙

双方都开启防火墙

lb01
[root@localhost ~]# systemctl start firewalld.service 
[root@localhost ~]# ip add show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:55:83:b7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.81.210/24 brd 192.168.81.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.81.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe55:83b7/64 scope link 
       valid_lft forever preferred_lft forever

lb02
[root@jxl ~]# systemctl start firewalld.service
[root@jxl ~]# ip add show dev ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:0c:29:46:66:34 brd ff:ff:ff:ff:ff:ff
    inet 192.168.81.220/24 brd 192.168.81.255 scope global ens33
       valid_lft forever preferred_lft forever
    inet 192.168.81.100/32 scope global ens33
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe46:6634/64 scope link 
       valid_lft forever preferred_lft forever

到此会发现双方都抢占了虚拟ip

用wireshare抓包也可以看到，都在争宠

3.解决由于防火墙裂脑的脚本

在备上编写检测脚本，测试如能ping通并且备节点还有VIP的话则认为产生了裂脑

[root@jxl ~]# vim check_keepalived.sh 
lb01=192.168.81.210
vip=192.168.81.100
while true;do
    vip_count=`ip add | grep $vip | wc -l`
    ping -c 2 -W 3 $lb01 &>/dev/null
    if [ $? -eq 0 -a $vip_count -eq 1 ];then
        echo -e "\\033[31mkeepalived is split brain.waringing...\\033[0m"
    else
        echo -e "\\033[32mkeepalived is ok....\\033[0m"
    fi
sleep 5s
done 

4.模拟故障案例2-nginx

在master上停掉nginx即可

5.解决nginx异常而导致主备不能切换

主节点的nginx宕掉后，网页无法打开，虚拟ip也不会漂移
这时我们就需要编写一个检测nginx状态的脚本，并在keepalived中引入
主备keepalive服务器都需要操作
1.编写脚本
[root@localhost ~]# vim check_nginx.sh
#/bin/bash
#检查nginx
while true;do
        nginx_proc_count=`ps -C nginx --no-header | wc -l`
        if [ $nginx_proc_count -eq 0 ];then
                systemctl restart nginx
                if [ $? -eq 0 ];then
                        echo -e "\\033[33mnginx starting....\\033[0m"
                fi
                sleep 5
                nginx_proc_count=`ps -C nginx --no-header | wc -l`
                if [ $nginx_proc_count -eq 0 ];then
                        systemctl stop keepalived
                        echo -e "\\033[33mnginx is error,stop keepalived....\\033[0m"
                        exit 1
                fi
        fi
        sleep 5
done

2.在keepalived中引入脚本
主备keepalive服务器都需要操作
[root@jxl ~]# vim /etc/keepalived/keepalived.con
global_defs 
    router_id lb02


vrrp_script check_web 
    script "/root/check_nginx.sh"   


vrrp_instance VI_1 
    state BACKUP
    interface ens33
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication 
        auth_type PASS
        auth_pass 1111
    
    virtual_ipaddress 
       192.168.81.100
    

    track_script 
        check_web
    

[root@localhost keepalived]# systemctl restart keepalived