Keepalived+LVS(dr)高可用负载均衡集群的实现

Posted 2020-09-18 PyGoer

一 环境介绍

1.操作系统
CentOS Linux release 7.2.1511 (Core)

2.服务
keepalived+lvs双主高可用负载均衡集群及LAMP应用
keepalived-1.2.13-7.el7.x86_64
ipvsadm-1.27-7.el7.x86_64
httpd-2.4.6-45.el7.centos.x86_64
mariadb-5.5.52-1.el7.x86_64
php-5.4.16-42.el7.x86_64

二 原理及拓扑图

1.vrrp协议
vrrp(Virtual Redundant Routing Protocol)协议：
在现实的网络环境中，两台需要通信的主机大多数情况下并没有直接的物理连接。对于这样的情况，它们之间路由怎样选择？主机如何选定到达目的主机的下一跳路由，这个问题通常的解决方法有两种：
 在主机上使用动态路由协议(RIP、OSPF等)
 在主机上配置静态路由
很明显，在主机上配置动态路由是非常不切实际的，因为管理、维护成本以及是否支持等诸多问题。配置静态路由就变得十分流行，但路由器(或者说默认网关default gateway)却经常成为单点故障。VRRP的目的就是为了解决静态路由单点故障问题，VRRP通过一竞选(election)协议来动态的将路由任务交给LAN中虚拟路由器中的某台VRRP路由器。

2.keepalived简介
Keepalived 是一个基于VRRP协议来实现的LVS服务高可用方案，可以利用其来避免单点故障。一个LVS服务会有2台服务器运行Keepalived，一台为主服务器（MASTER），一台为备份服务器（BACKUP），但是对外表现为一个虚拟IP，主服务器会发送特定的消息给备份服务器，当备份服务器收不到这个消息的时候，即主服务器宕机的时候，备份服务器就会接管虚拟IP，继续提供服务，从而保证了高可用性。Keepalived是VRRP的完美实现。

3.lvs-dr
Direct Routing，直接路由,通过为请求报文重新封装一个MAC首部进行转发，源MAC是DIP所在的接口的MAC，目标MAC是某挑选出的RS的RIP所在接口的MAC地址；源IP/PORT，以及目标IP/PORT均保持不变。  
Director和各RS都得配置使用VIP； 
(1) 确保前端路由器将目标IP为VIP的请求报文发往Director；
 (a) 在前端网关做静态绑定；
 (b) 在RS上使用arptables；
 (c) 在RS上修改内核参数以限制arp通告及应答级别；
  arp_announce
  arp_ignore
(2) RS的RIP可以使用私网地址，也可以是公网地址；RIP与DIP在同一IP网络；RIP的网关不能指向DIP，以确保响应报文不会经由Director； 
(3) RS跟Director要在同一个物理网络；
(4) 请求报文要经由Director，但响应不能经由Director，而是由RS直接发往Client；
(5) 不支持端口映射。

4.IP分配
VIP1:172.18.67.66
VIP2:172.18.67.88
DIP1:172.18.67.13
DIP2:172.18.67.14
RIP1:172.18.67.11
RIP2:172.18.67.12
CIP:172.18.67.3

5.拓扑图

三 keepalived配置

1.安装keepalived

[[email protected] ~]# yum install -y keepalived
[[email protected] ~]# yum install -y keepalived

 

2.高可用的ipvs双主集群配置  
第一个节点：

[[email protected] ~]# cd /etc/keepalived/
[[email protected] keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]                  　　　　　　　　　　　　　　#接受通知的邮件地址
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node2                 　　　　　　　　　　　　　　 #路由节点
vrrp_mcast_group4 224.0.67.67              　　　　　　　　#多播地址，范围224.0.0.0~239.255.255.255
}
vrrp_instance myr1 {
state MASTER
interface eno16777736                　　　　　　　　　　　　#网卡接口
virtual_router_id 167                　　　　　　　　　　　　#虚拟路由ID号，0~255
priority 100                  　　　　　　　　　　　　　　　　#优先级，MASTER比BACKUP优先级高
advert_int 1
authentication {
 auth_type PASS
 auth_pass f1bf7fda
}
virtual_ipaddress {
 172.18.67.66/16 dev eno16777736 label eno16777736:0
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"         #调用通知脚本
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance myr2 {
state BACKUP
interface eno16777736
virtual_router_id 168
priority 98
advert_int 1
authentication {
 auth_type PASS
 auth_pass f2bf7ade
}
virtual_ipaddress {
 172.18.67.88/16 dev eno16777736 label eno16777736:1
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 172.18.67.66 80 {              #VIP
delay_loop 2
lb_algo wrr                  　　　　　　　　　　#lvs负载均衡调度算法
lb_kind DR                   　　　　　　　　　　#负载均衡类型
protocol TCP                  　　　　　　　　　 #传输协议
sorry_server 127.0.0.1 80               　　　 #localhost
real_server 172.18.67.11 80 {              　  #后端RIP
 weight 1                  　　　　　　　　　　　 #调度权重 
 HTTP_GET {                  　　　　　　　　　　#http请求方式
 url {
  path /
  status_code 200               　　　　　　　　#状态码
 }
 connect_timeout 2                　　　　　　  #连接超时
 nb_get_retry 3
 delay_before_retry 3
 }
}
real_server 172.18.67.12 80 {
 weight 1
 HTTP_GET {
  url {
   path /
   status_code 200
  }
  connect_timeout 2
  nb_get_retry 3
  delay_before_retry 3
  }
 }
}

 

第二个节点：

[[email protected] ~]# cd /etc/keepalived/
[[email protected] keepalived]# vim keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[email protected]
}
notification_email_from [email protected]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node3
vrrp_mcast_group4 224.0.67.67
}
vrrp_instance myr1 {
state BACKUP
interface eno16777736
virtual_router_id 167
priority 98
advert_int 1
authentication {
 auth_type PASS
 auth_pass f1bf7fda
}
virtual_ipaddress {
 172.18.67.66/16 dev eno16777736 label eno16777736:0
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
vrrp_instance myr2 {
state MASTER
interface eno16777736
virtual_router_id 168
priority 100
advert_int 1
authentication {
 auth_type PASS
 auth_pass f2bf7ade
}
virtual_ipaddress {
 172.18.67.88/16 dev eno16777736 label eno16777736:1
}
track_interface {
 eno16777736
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
virtual_server 172.18.67.88 80 {
delay_loop 2
lb_algo wrr
lb_kind DR
protocol TCP
sorry_server 127.0.0.1 80
real_server 172.18.67.11 80 {
 weight 1
 HTTP_GET {
 url {
  path /
  status_code 200
 }
 connect_timeout 2
 nb_get_retry 3
 delay_before_retry 3
 }
}
real_server 172.18.67.12 80 {
 weight 1
 HTTP_GET {
  url {
   path /
   status_code 200
  }
  connect_timeout 2
  nb_get_retry 3
  delay_before_retry 3
  }
 }
}

3.邮件通知脚本
当双主高可用集群主备切换时可通过邮件通知管理员，此时在配置文件中可自动调用实现编辑好的脚本

[[email protected] ~]# vim notify.sh
#!/bin/bash
#
contact=‘[email protected]‘
notify() {
 mailsubject="$(hostname) to be $1, vip floating"
 mailbody="$(date +‘%F %T‘): vrrp transition, $(hostname) changed to be $1"
 echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
 notify master
 ;;
backup)
 notify backup
 ;;
fault)
 notify fault
 ;;
*)
 echo "Usage: $(basename $0) {master|backup|fault}"
 exit 1
 ;;
esac

节点二同样配置。

四 后端RS服务器的配置

1.配置LAMP环境

[[email protected] ~]# yum install httpd mariadb-server php -y
[[email protected] ~]# yum install httpd mariadb-server php -y

 

2.简单编辑测试网页

[[email protected] ~]# echo "RS1:172.18.67.11" > /var/www/html/index.html
[[email protected] ~]# echo "RS2:172.18.67.12" > /var/www/html/index.html

 

3.修改RS内核参数
dr模型中，各主机上均需要配置VIP，解决地址冲突的方式有三种：
(1) 在前端网关做静态绑定；
(2) 在各RS使用arptables；
(3) 在各RS修改内核参数，来限制arp响应和通告的级别；
限制响应级别：arp_ignore
 0：默认值，表示可使用本地任意接口上配置的任意地址进行响应；
 1: 仅在请求的目标IP配置在本地主机的接收到请求报文接口上时，才给予响应；
限制通告级别：arp_announce
 0：默认值，把本机上的所有接口的所有信息向每个接口上的网络进行通告；
 1：尽量避免向非直接连接网络进行通告；
 2：必须避免向非本网络通告。
可通过编辑脚本实现：

[[email protected] ~]# vim dr.sh
#!/bin/bash
#
case $1 in
start)
 echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
 echo 1 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
 echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
 echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
 ;;
stop)
 echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
 echo 0 > /proc/sys/net/ipv4/conf/eth0/arp_ignore
 echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
 echo 0 > /proc/sys/net/ipv4/conf/eth0/arp_announce
 ;;
*) 
 echo "Usage $(basename $0) start|stop"
 exit 1
 ;;
esac

同理第二个RS需同样配置

4.添加路由
节点一：

[[email protected] ~]# ifconfig lo:0 172.18.67.66 netmask 255.255.255.255 broadcast 172.18.67.66 up
[[email protected] ~]# ifconfig lo:1 172.18.67.88 netmask 255.255.255.255 broadcast 172.18.67.88 up
[[email protected] ~]# route add -host 172.18.67.66 dev lo:0
[[email protected] ~]# route add -host 172.18.67.88 dev lo:1

节点二：

[[email protected] ~]# ifconfig lo:0 172.18.67.88 netmask 255.255.255.255 broadcast 172.18.67.88 up
[[email protected] ~]# ifconfig lo:1 172.18.67.66 netmask 255.255.255.255 broadcast 172.18.67.66 up
[[email protected] ~]# route add -host 172.18.67.88 dev lo:0
[[email protected] ~]# route add -host 172.18.67.66 dev lo:1

 

五 测试

1.启动服务

RS:

[[email protected] ~]# systemctl start httpd
[[email protected] ~]# systemctl start httpd

 

节点一：

[[email protected] ~]# systemctl start keepalived.service
[[email protected] ~]# systemctl status -l  keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-05-14 01:19:27 CST; 17s ago
  Process: 2120 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2121 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─2121 /usr/sbin/keepalived -D
           ├─2122 /usr/sbin/keepalived -D
           └─2123 /usr/sbin/keepalived -D
May 14 01:19:29 inode2 Keepalived_vrrp[2123]: Opening script file /etc/keepalived/notify.sh
May 14 01:19:29 inode2 Keepalived_healthcheckers[2122]: Netlink reflector reports IP 172.18.67.66 added
May 14 01:19:31 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Transition to MASTER STATE
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Entering MASTER STATE
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) setting protocol VIPs.
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
May 14 01:19:32 inode2 Keepalived_vrrp[2123]: Opening script file /etc/keepalived/notify.sh
May 14 01:19:32 inode2 Keepalived_healthcheckers[2122]: Netlink reflector reports IP 172.18.67.88 added
May 14 01:19:34 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr1) Sending gratuitous ARPs on eno16777736 for 172.18.67.66
May 14 01:19:37 inode2 Keepalived_vrrp[2123]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
[[email protected] ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:8b:08:6f brd ff:ff:ff:ff:ff:ff
    inet 172.18.67.13/16 brd 172.18.255.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 172.18.67.66/16 scope global secondary eno16777736:0
       valid_lft forever preferred_lft forever
    inet 172.18.67.88/16 scope global secondary eno16777736:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe8b:86f/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever

 

节点二：

[[email protected] ~]# systemctl start keepalived.service
[[email protected] ~]# systemctl status -l  keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2017-05-14 01:20:25 CST; 6s ago
  Process: 2110 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2111 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─2111 /usr/sbin/keepalived -D
           ├─2112 /usr/sbin/keepalived -D
           └─2113 /usr/sbin/keepalived -D
May 14 01:20:25 inode3 Keepalived_vrrp[2113]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Transition to MASTER STATE
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:26 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Received lower prio advert, forcing new election
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Entering MASTER STATE
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) setting protocol VIPs.
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: VRRP_Instance(myr2) Sending gratuitous ARPs on eno16777736 for 172.18.67.88
May 14 01:20:27 inode3 Keepalived_vrrp[2113]: Opening script file /etc/keepalived/notify.sh
May 14 01:20:27 inode3 Keepalived_healthcheckers[2112]: Netlink reflector reports IP 172.18.67.88 added
[[email protected] ~]# ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno16777736: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:0c:29:78:24:c3 brd ff:ff:ff:ff:ff:ff
    inet 172.18.67.14/16 brd 172.18.255.255 scope global eno16777736
       valid_lft forever preferred_lft forever
    inet 172.18.67.88/16 scope global secondary eno16777736:1
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe78:24c3/64 scope link tentative dadfailed 
       valid_lft forever preferred_lft forever

 

2.访问测试

[[email protected] ~]# curl http://172.18.67.66
RS2:172.18.67.12
[[email protected] ~]# curl http://172.18.67.66
RS1:172.18.67.11
[[email protected] ~]# curl http://172.18.67.66
RS2:172.18.67.12
[[email protected] ~]# curl http://172.18.67.66
RS1:172.18.67.11
[[email protected] ~]# curl http://172.18.67.88
RS2:172.18.67.12
[[email protected] ~]# curl http://172.18.67.88
RS1:172.18.67.11
[[email protected] ~]# curl http://172.18.67.88
RS2:172.18.67.12
[[email protected] ~]# curl http://172.18.67.88

高可用负载集群成功搭建完成。

3.模拟一台web服务器宕机
将RS1的httpd服务停掉，并再此访问：

[[email protected]:~]# systemctl stop httpd
[[email protected] ~]# curl http://172.18.67.66
RS2:172.18.67.12
[[email protected] ~]# curl http://172.18.67.66
RS2:172.18.67.12
[[email protected] ~]# curl http://172.18.67.88
RS2:172.18.67.12
[[email protected] ~]# curl http://172.18.67.88

发现照样可以访问服务器，实际环境中两台RS的内容应该是一模一样的，在这里我为了以示区别将两台内容编辑成不同。

4.模拟一台高可用负载集群宕机
将VS1的keepalived服务关闭并测试：

[[email protected] ~]# systemctl stop keepalived.service
[[email protected] ~]# curl http://172.18.67.88
RS1:172.18.67.11
[[email protected] ~]# curl http://172.18.67.88
RS2:172.18.67.12
[[email protected] ~]# curl http://172.18.67.88
RS1:172.18.67.11
[[email protected] ~]# curl http://172.18.67.88
RS2:172.18.67.12

 

我们发现即使高可用负载均衡集群中的某一个主机宕机了，我们任然可以通过其中的一个IP访问web服务器，体现出了高可用的实用性，并且在访问中lvs调度器将客户端请求按设置的权重分别向后端的服务器实现调度。