解决Lvs+keepalived出现双VIP，即脑裂现象
Posted
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了解决Lvs+keepalived出现双VIP，即脑裂现象相关的知识，希望对你有一定的参考价值。
参考技术A 如果看到两台机器都绑定了vip那么就是出现了双vip的现象，即脑裂现象。例如下面LVS+keepalived生产环境实现电信联通双VIP(DR模式)
参考:
http://duyunlong.blog.51cto.com/1054716/1118447
http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-DR.html
http://www.cnblogs.com/Eivll0m/p/4580826.html
防火墙配置参考:http://wgkgood.blog.51cto.com/1192594/1102870
server全部为centos6.8
[[email protected]_MASTER keepalived]# uname -a Linux iiw_MASTER 2.6.32-642.4.2.el6.x86_64 #1 SMP Tue Aug 23 19:58:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
一:需求.由于前期经费有限,说白了就是没钱.
只能使用单主机做keepalived master,后端三台nginx提供web服务.开发80,443端口.
所需的ip为(安全原因ip都未虚构):
| 电信IP | 联通IP | |
| VIP | 10.10.10.214 | 36.10.10.86 |
| keepaliver master主机 | 10.10.10.213 | 36.10.10.85 |
| realserver1主机 | 10.10.10.215 | 36.10.10.87 |
| realserver2主机 | 10.10.10.216 | 36.10.10.88 |
| realserver3主机 | 10.10.10.217 | 36.10.10.89 |
二.流程
masterserver主机(ip:10.10.10.213),安装lvs+keeplived.开启ip转发功能.
[[email protected]_MASTER keepalived]# cat /etc/sysctl.conf net.ipv4.ip_forward = 1
realserver安装nginx,并开启服务.略
masterserver,realserver 防火墙开发80,443端口.开放虚拟路由广播:
iptables -A INPUT -p tcp --dport 80 -j ACCEPT iptables -A INPUT -p tcp --dport 443 -j ACCEPT
4.后端realserver,VIP绑定脚本文件为:
[[email protected]_WEB1 ~]$ cat /etc/init.d/realserver.sh #!/bin/bash SNS_VIP=10.10.10.214 #电信vip SNS_VIP1=36.10.10.86 #联通vip . /etc/rc.d/init.d/functions case "$1" in start) ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP up /sbin/route add -host $SNS_VIP dev lo:0 ifconfig lo:1 $SNS_VIP1 netmask 255.255.255.255 broadcast $SNS_VIP1 up /sbin/route add -host $SNS_VIP1 dev lo:1 echo "1" > /proc/sys/net/ipv4/conf/lo/arp_ignore echo "2" > /proc/sys/net/ipv4/conf/lo/arp_announce echo "1" > /proc/sys/net/ipv4/conf/all/arp_ignore echo "2" > /proc/sys/net/ipv4/conf/all/arp_announce sysctl -p > /dev/null 2>&1 echo "RealServer Start OK" ;; stop) ifconfig lo:0 down route del $SNS_VIP >/dev/null 2>&1 ifconfig lo:1 down route del $SNS_VIP1 >/dev/null 2>&1 echo "0" > /proc/sys/net/ipv4/conf/lo/arp_ignore echo "0" > /proc/sys/net/ipv4/conf/lo/arp_announce echo "0" > /proc/sys/net/ipv4/conf/all/arp_ignore echo "0" > /proc/sys/net/ipv4/conf/all/arp_announce echo "RealServer Stoped" ;; *) echo "Usage: $0 {start|stop}" exit 1 esac exit 0
开启脚本:
[[email protected]_WEB1 ~]$ /etc/init.d/realserver.sh start
检测vip是否存在:
[[email protected]_WEB1 ~]$ ifconfig ............................ lo:0 Link encap:Local Loopback inet addr:10.10.10.214 Mask:255.255.255.255 UP LOOPBACK RUNNING MTU:65536 Metric:1 lo:1 Link encap:Local Loopback inet addr:36.10.10.86 Mask:255.255.255.255 UP LOOPBACK RUNNING MTU:65536 Metric:1
可见我们所需的两个VIP已经绑定在网卡l0:0,l0:1上.
5.masterserver上keepalived的配置:
[[email protected]_MASTER keepalived]# cat /etc/keepalived/keepalived.conf ! Configuration File for keepalived global_defs { notification_email { [email protected] } notification_email_from [email protected] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_MASTER } #电信vip设置. vrrp_instance VI_1 { state MASTER #注意此住为master状态 interface eth0 #检测eth0网卡 virtual_router_id 51 #注意id是51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.10.10.214 } } #联通VIP设置 vrrp_instance VI_2 { state MASTER #注意此住为master状态 interface eth0 #检测eth0网卡 virtual_router_id 52 #复制的时候注意此处id是52,不要跟电信VIP相同 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 36.10.10.86 } } virtual_server 10.10.10.214 80 { delay_loop 6 lb_algo wrr lb_kind DR nat_mark 255.255.255.0 protocol TCP real_server 10.10.10.215 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } real_server 10.10.10.216 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } real_server 10.10.10.217 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } } virtual_server 10.10.10.214 443 { delay_loop 6 lb_algo wrr lb_kind DR nat_mark 255.255.255.0 protocol TCP real_server 10.10.10.215 443 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 443 } } real_server 10.10.10.216 443 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 443 } } real_server 10.10.10.217 443 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 443 } } } virtual_server 36.10.10.86 80 { delay_loop 6 lb_algo wrr lb_kind DR nat_mark 255.255.255.0 protocol TCP real_server 36.10.10.87 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } real_server 36.10.10.88 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } real_server 36.10.10.89 80 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 80 } } } virtual_server 36.10.10.86 443 { delay_loop 6 lb_algo wrr lb_kind DR nat_mark 255.255.255.0 protocol TCP real_server 36.10.10.87 443 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 443 } } real_server 36.10.10.88 443 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 443 } } real_server 36.10.10.89 443 { weight 1 TCP_CHECK { connect_timeout 3 nb_get_retry 3 delay_before_retry 3 connect_port 443 } } }
在此,我将两个VIP分别绑定在两个vrrp_instance上,并且每个vrrp_instance都设置为master.然后启动keepalived
[[email protected]_MASTER keepalived]# service keepalived start
检测vip:
[[email protected]_MASTER keepalived]# ip add 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 3c:4a:92:e0:d1:30 brd ff:ff:ff:ff:ff:ff inet 10.10.10.213/27 brd 10.10.10.223 scope global eth0 inet 36.10.10.85/27 brd 36.10.10.95 scope global eth0:0 inet 10.10.10.214/32 scope global eth0 inet 36.10.10.86/32 scope global eth0 inet6 fe80::3e4a:92ff:fee0:d130/64 scope li
可见我们所需的vip已经绑定在eth0上:
inet 10.10.10.214/32 scope global eth0 inet 36.10.10.86/32 scope global eth0
6.使用ipvsadm检测后端realserver是否ok轮询:
[[email protected]_MASTER keepalived]# ipvsadm -Ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 36.10.10.86:80 wrr -> 36.10.10.87:80 Route 1 0 18 -> 36.10.10.88:80 Route 1 0 17 -> 36.10.10.89:80 Route 1 0 17 TCP 36.10.10.86:443 wrr -> 36.10.10.87:443 Route 1 0 2 -> 36.10.10.88:443 Route 1 0 5 -> 36.10.10.89:443 Route 1 0 3 TCP 10.10.10.214:80 wrr -> 10.10.10.215:80 Route 1 7 25 -> 10.10.10.216:80 Route 1 11 25 -> 10.10.10.217:80 Route 1 7 25 TCP 10.10.10.214:443 wrr -> 10.10.10.215:443 Route 1 4 3 -> 10.10.10.216:443 Route 1 1 6 -> 10.10.10.217:443 Route 1 1 4
通过这个命令可见,后端三台realserver的nginx 80,443端口被完全识别.
7.通过hosts绑定网站为联通VIP36.10.10.86.通过AB软件测试:
[email protected]:/home/huwei# ulimit -SHn 51200 [email protected]:/home/huwei# ab -n 10000 -c 100 "http://www.iiii111.com"
测试,通过http访问网站1w次,使用watch "lpvsadm"查看会不会平均分布到后端realserver
[[email protected]_MASTER keepalived]# watch "ipvsadm"
Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 36.10.10.86:http wrr -> 36.10.10.87:http Route 1 0 3354 -> 36.10.10.88:http Route 1 0 3354 -> 36.10.10.89:http Route 1 0 3354
换https测试:
[email protected]:/home/huwei# ab -n 10000 -c 100 "https://www.iiii111.com"
结果:
TCP 36.10.10.86:https wrr -> 36.10.10.87:https Route 1 0 3366 -> 36.10.10.88:https Route 1 0 3364 -> 36.10.10.89:https Route 1 1 3366
可见,轮询成功.
换hosts绑定电信vip再测也正常.
到此结束.
总结:1.keepalived后端的nginx 80,443端口不要写错.
2.防火墙配置要正确.
3.后端服务器vip脚本先记得要启动,并检查vip ip有没有绑定好.
本文出自 “海底两万里” 博客，请务必保留此出处http://huwei555.blog.51cto.com/1347480/1865124
以上是关于解决Lvs+keepalived出现双VIP，即脑裂现象的主要内容，如果未能解决你的问题，请参考以下文章