LVS+Keepalived高可用负载均衡架构原理及配置

Posted 2020-10-26

1、keepalived 介绍
2、keepalived 优缺点
3、keepalived 应用场景
4、keepalived 安装配置
5、keepalived+lvs 高可用
6、keepalived+nginx 高可用
7、keepalived 切换原理
8、性能优化
9、常见故障

一、keepalived 介绍

1.keepalived 定义
keepalived是一个基于VRRP(virtual route redundent protocol)协议来实现的LVS服务高可用方案，可以利用其来避免单点故障。
一个LVS服务会有2台 服务器运行keepalived，一台为主服务器，一台为备服务器，但对外表现一个虚拟IP。
主服务会发送特定的消息给备服务器，当备服务器无法接收到主服务器的消息时，即认为主服务器宕机，备服务器会接管主服务器的VIP，继续提供服务，从而保证高可用性。

2.VRRP协议介绍
VRRP的目的就是为了解决静态路由单点故障问题，VRRP通过竞选协议来动态的将路由任务交给LAN中虚拟路由器中的某台VRRP路由器。
当有多台VRRP时，通过竞选，只有一台能成为master，master能拿到VIP，来转发送给网关的地址和包响应arp请求。
VRRP通过协议来竞选master，协议报文都是通过IP多播包形式发送的，对外都使用同一个mac地址，客户端主机不会因为master的更改来自己的路由配置，对于客户端来说master的主从切换是透明的。
正常情况下 master会一直发送vrrp通告信息，backup不会抢占master，除非backup的优先级比master更高，当master的宕机，优先级最高的backup在>1s的时间内进行抢占。

二、keepalived 优点缺点

优点： 轻量级、配置简单
缺点：不能实现服务状态级别的高可用

HA与LB的区别
HA：实现服务的高可用
LB：实现流量入口的最大化

三、keepalived应用场景

keepalived中实现轻量级的高可用，一般用于前端高可用，且不需要共享存储，一般常用语两个节点的高可用；
常见组合：
lvs+keepalived
nginx+keepalived
haproxy+keepalived

与heartbeat、corosync比较
heartbeat与corosync一般用于服务的高可用，且需要共享存储，一般用于多节点的高可用。

四、keepalived 安装配置

4.1软件环境安装准备

系统：Centos 6.5
yum源：本地yum源
地址规划：
Keepalived VIP： 192.168.200.139
LVS主机2台： 192.168.200.132 192.168.200.133
real-server 主机2台： 192.168.200.134 192.168.200.135

4.2 LVS+Keepalived 安装

从centos6.3以后keeplive收录到base
#yum -y install keeplived ipvsadm //使用本地或者网络Yum源进行安装
#yum info keeplived //查看安装的信息
#rpm -ql keeplived ipvsadm //查看是否安装完成

4.3 Keepalived配置文件介绍

#vim /etc/keeplived/keeplived.conf //配置文件
配置文件组成部分：
global_configuration 全局配置段
vrrpd configuration vrrp配置进程
vrrp instance
vrrp synchonization group
lvs configuration lvs配置段
shell>man keepalived.conf //配置手册

4.4 Keepalived主备高可用模式

4.4.1 LVS主节点（192.168.200.132）配置

创建一个实例，实现VIP 192.168.200.139
shell>vim /etc/keeplived/keeplived.conf //默认是开启抢占模式的
global_defs { //全局配置段，
notification_email { //定义接收邮件br/>[email protected]
}
notification_email_from root
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL //集群组ID
vrrp_mcast_group //定义广播地址用于心跳通信，双主不需要定义，双主会影响
}

vrrp_script chk_xxx { //定义脚本策略，用户在线切换vip
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
interval 1 //2秒发一次检查
weight -2 //down文件存在优先级减去2
}

vrrp_instance VI_1 { //keepalived实例段
state MASTER //keepalived主节点
interface eth0 //通信端口
virtual_router_id 51 //定义虚拟路由id，同一个实例必须一样，可以定义多个实例
priority 102 //定义实例优先级，越大越优先，0-255
advert_int 1 //检查时间间隔，不能太长不能太短
nopreempt //设置非抢占
authentication { //加密验证
auth_type PASS
auth_pass 1111
}

virtual_ipaddress { //定义虚拟VIP
192.168.200.139 dev eth0 label eth0:0
}

track_script {
chk_xxx //调用脚本进行检查
}
}

real_server 192.168.200.134 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
real_server 192.168.200.135 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
sorry_server 127.0.0.1 80

4.4.2 LVS备节点（192.168.200.133）配置

shell>vim /etc/keeplived/keeplived.conf //默认是开启抢占模式的
global_defs { //全局配置段，
notification_email { //定义接收邮件br/>[email protected]
}
notification_email_from root
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL //集群组ID
vrrp_mcast_group //定义广播地址用于心跳通信，双主不需要定义，双主会影响
}

vrrp_script chk_xxx { //定义脚本策略，用户在线切换vip
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"
interval 1 //2秒发一次检查
weight -2 //down文件存在优先级减去2
}

vrrp_instance VI_1 { //keepalived实例段
state BACKUP //keepalived备节点
interface eth0 //通信端口
virtual_router_id 51 //定义虚拟路由id，同一个实例必须一样，可以定义多个实例
priority 100 //定义实例优先级，越大越优先，0-255
advert_int 1 //检查时间间隔，不能太长不能太短
nopreempt //设置非抢占
authentication { //加密验证
auth_type PASS
auth_pass 1111
}

virtual_ipaddress { //定义虚拟VIP
192.168.200.139 dev eth0 label eth0:0
}

track_script {
chk_xxx //调用脚本进行检查
}

real_server 192.168.200.134 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
real_server 192.168.200.135 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
sorry_server 127.0.0.1 80

}

4.4.3 解析

【这就是vrrp_script、track_script脚本基本工作机制；】
【可利用这样的脚本来判断http服务是否ok，实现出现故障时，地址自动浮动到另一节点上继续提供服务；】
【要判断一个服务是否在线，脚本应写的越简单越好；】
shell> killall -0 httpd
httpd: no process found
shell> echo $?
【显示为1】
在实例上添加一个vrrp_script chk_httpd策略，然后调用在实例中，即可实现对网站httpd程序的监控
配置完成后，会自动在ipvsadm 中添加一个虚拟主机和2个real-server主机。
shell> ipvsadm -Ln 进行查看

4.4.4 日志定义

#vim /etc/sysconfig/keepalived 添加日志
KEEPALIVED_OPSTION "D -S 3"

vim /etc/rsyslog.conf
local3.* /var/log/keepalived.log

#systemctl restart rsyslog.service
#systemctl restart keepalived.service

4.4.5 邮件通知脚本

邮件通知脚本
shell> vim /etc/keepalived/notify.shbr/>#!/bin/bash
vip=192.168.200.139
contact=‘[email protected]‘
{ notify()
mailsubject="hostname to be $1: $vip floating"
mailbody="date‘ +%F %H:%M:%S‘ : vrrp transition, hostname changed to be $1"
echo $mailbody | mail -s "$mailsubject" $contact
}
case "$1" in
master)
notify master
exit 0
;;
backup)
notify backup
exit 0
;;
fault)
notify fault
exit 0
;;
*)
echo ‘usage: basename $0 {master|backup|fault}‘
exit 1
;;
esac
}

shell>chmod o+x /etc/keepalived/notify.sh

在配置文件中建立2个实例，一主一备，当域名有2个A记录是，实现双主调度
注意不要定义组播地址：
vrrp_instance VI_2
state BACKUP
interface eth0:1
vritual_router_id 61 //不能与实例1一样
priority 99 //低于master
advert_int 1
authentication //不能与实例一一样

virtual_ipaddress {
192.168.200.139/16 dev eth0 label eth0:0
}定义一个ipvs集群

virtual_server 192.168.200.139 80 { //vip
delay_loop 6 //检查失败转6圈
lb_algo rr //调度方式
lb_kind DR //lvs类型
nat_mask 255.255.255.0 //掩码
#persistence_timeout 50 //持久连接
protocol TCP
sorry_server 127.0.0.1 //无法提供访问返回页面
real_server 192.168.200.134 80 {
weight 1
HTTP_GET {
url {
path / #请求根，可以给多个url
status_code 200 #返回状态值
}
connect_timeout 2 #连接超时
nb_get_retry 3 #失败重试
delay_before_retry 1
}
}
sorry_server 127.0.0.1 //故障提示

抓包
#tcpdump -i eth0 -nn host 192.168.200.133

健康状态检查，TCP_CHECK精度没有HTTP_GET高,SSL_GET(https)

real_server 192.168.200.134 80 {
weight 1
TCP_CHECK {
connect_timeout 3
}

五、完整的基于keepalived的lvs实验

(双主只需多加入一个实例)
node1配置
#vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {br/>[email protected]
}
notification_email_from root
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node132
}

vrrp_script chk_xxx {
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit0"
interval 1
weight -20
}

vrrp_script chk_httpd {
script "killall -0 httpd"
interval 2
weight -5
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

virtual_ipaddress {
192.168.200.139 dev eth0 label eth0:0
}

track_script {
chk_xxx
chk_httpd
}

notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}

virtual_server 192.168.200.139 80 {
delay_loop 6
lb_algo rr
lb_kind DR
nat_mask 255.255.255.0
protocol TCP
real_server 192.168.200.134 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
real_server 192.168.200.135 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
sorry_server 127.0.0.1 80

node2配置
! Configuration File for keepalived

global_defs {
notification_email {br/>[email protected]
}
notification_email_from root
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node132
}

vrrp_script chk_xxx {
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit0"
interval 1
weight -20
}

vrrp_script chk_httpd {
script "killall -0 httpd"
interval 2
weight -5
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 99
nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

virtual_ipaddress {
192.168.200.139 dev eth0 label eth0:0
}

track_script {
chk_xxx
chk_httpd
}

notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}

virtual_server 192.168.200.139 80 {
delay_loop 6
lb_algo rr
lb_kind DR
nat_mask 255.255.255.0
protocol TCP
real_server 192.168.200.134 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
real_server 192.168.200.135 80 {
weight 1
HTTP_GET {
url {
path /
status_code 200
}
connect_timeout 2
nb_get_retry 3
delay_before_retry 1
}
}
sorry_server 127.0.0.1 80

通过脚本实现real server 配置
real server 1-2配置
shell>vim /etc/rc.d/init.d/realserver.sh
#!/bin/bash
#description: Config realserver lo and apply noarp

SNS_VIP=192.168.200.139

/etc/rc.d/init.d/functions

case "$1" in
start)
ifconfig lo:0 $SNS_VIP netmask 255.255.255.255 broadcast $SNS_VIP
/sbin/route add -host $SNS_VIP dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
sysctl -p >/dev/null 2>&1
echo "RealServer Start OK" /bin/true

   ;;

stop)
ifconfig lo:0 down
route del $SNS_VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "RealServer Stoped" /bin/true
;;
*)
echo "Usage: $0 {start|stop}" /bin/false
exit 1
esac

exit 0

六、keepalived集群Nginx负载均衡

（双主配置）
正常安装nginx···
[[email protected] keepalived]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived

global_defs {
notification_email {br/>[email protected]
}
notification_email_from root
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node133
}

vrrp_script chk_xxx {
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit0" #当该目录有down文件就切换
interval 1
weight -20
}

vrrp_script chk_httpd {
script "killall -0 nginx"
interval 2
weight -5
}

vrrp_instance VI_11 {
state BACKUP
interface eth0
virtual_router_id 51
priority 80
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.200.139 dev eth0 label eth0:0
}
track_script {
chk_xxx
chk_httpd
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}

vrrp_instance VI_22 {
state MASTER
interface eth0
virtual_router_id 61
priority 110
advert_int 1
authentication {
auth_type PASS
auth_pass 2222
}

virtual_ipaddress {
192.168.200.137 dev eth0 label eth0:1
}

track_script {
chk_xxx
chk_httpd
}

notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
sorry_server 127.0.0.1

[[email protected] keepalived]# vim /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {
notification_email {br/>[email protected]
}
notification_email_from root
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id node132
}

vrrp_script chk_xxx {
script "[[ -f /etc/keepalived/down ]] && exit 1 || exit0"
interval 1
weight -20
}

vrrp_script chk_httpd {
script "killall -0 nginx"
interval 2
weight -5
}

vrrp_instance VI_11 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

virtual_ipaddress {
192.168.200.139 dev eth0 label eth0:0
}

track_script {
chk_xxx
chk_httpd
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}

vrrp_instance VI_22 {
state BACKUP
interface eth0
virtual_router_id 61
priority 99
nopreempt
advert_int 1
authentication {
auth_type PASS
auth_pass 2222
}

virtual_ipaddress {
192.168.200.137 dev eth0 label eth0:1
}

track_script {
chk_xxx
chk_httpd
}
notify_master "/etc/keepalived/notify.sh master"
notify_backup "/etc/keepalived/notify.sh backup"
notify_fault "/etc/keepalived/notify.sh fault"
}
sorry_server 127.0.0.1

以上是关于LVS+Keepalived高可用负载均衡架构原理及配置的主要内容，如果未能解决你的问题，请参考以下文章

RedHat 7配置keepalived+LVS实现高可用的Web负载均衡

lvs+keepalived实现高可用负载均衡

集群------LVS+Keepalived高可用负载均衡群集

LVS+Keepalived实现负载均衡+高可用性

Keepalived负载均衡与高可用

LVS+keepalived高可用负载均衡集群部署 ----数据库的读写分离