Keepalived+LVS实战案例：单主架构实现WEB负载均衡及可用

Posted 2022-04-01 njsummer

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Keepalived+LVS实战案例：单主架构实现WEB负载均衡及可用相关的知识，希望对你有一定的参考价值。

1. 概述

Keepalived是通过vrrp 协议的实现高可用性，对网络比较了解的IT人，对这个技术应该非常熟悉了，早期核心交换机用来实现双机双线的标准协议，现在随着技术发展出现了更好的核心设备的双活技术，vrrp/hrrp慢慢被取代了，但目前在Linux主机类应用场景使用还比较广泛。它的原生设计目的为了解决 ipvs高可用性。

官网：http://keepalived.org/

功能：

基于vrrp协议完成地址漂移；
为vip地址所在的节点生成ipvs规则 (在配置文件中预先定义)；
为ipvs集群的各RS做健康状态检测；（ keepalived 可以搭配 LVS、haproxy等成为黄金组合，尤其是 keepalived + haproxy 在很多企业生产中使用）
基于脚本调用接口完成脚本中定义的功能，进而影响集群事务，以此支持nginx、haproxy等服务。

2. 工作原理及技术架构

Keepalived+LVS实战案例（一）：

官方文档：

https://keepalived.org/doc/
http://keepalived.org/documentation.html

用户空间核心组件：

vrrp stack：VIP消息通告；

checkers：监测real server；

system call：实现 vrrp 协议状态转换时调用脚本的功能；

SMTP：邮件组件；

IPVS wrapper：生成IPVS规则；

Netlink Reflector：网络接口；

WatchDog：监控进程；

控制组件：提供keepalived.conf 的解析器，完成Keepalived配置。
IO复用器：针对网络目的而优化的自己的线程抽象。
内存管理组件：为某些通用的内存管理功能（例如分配，重新分配，发布等）提供访问权限。

3. 拓扑结构及主机准备

Keepalived+LVS实战案例（一）：

# 五台主机
1  2台web服务器 ：
主机名：WebServer-IP17
CentOS 7.9
IP:192.168.250.17

主机名：WebServer-IP27
CentOS 7.9
IP:192.168.250.27

2  2台 keepalived 服务器 ：
主机名： KA-IP18
CentOS 8.4
IP:192.168.250.18/24
Keepalived v2.1.5 (07/13,2020)

主机名： KA-IP28
CentOS 8.4
IP:192.168.250.28/24
Keepalived v2.1.5 (07/13,2020)

3  1台client主机 ：
主机名：Client-IP172-8
CentOS 8.4
IP:172.16.0.8/24  NAT成192.168.250.254 访问192.168.250.X网段

4. 后端WEB服务器准备

4.1 安装Apache httpd 完成基本的WWW配置

# 基础环境包括CentOS操作系统的优化、防火墙关闭、同步时间等都要做好，我们按照规划的架构图对四台服务器进行分组并重新命名
# 修改服务器名称
[root@centos79 <sub>]# hostnamectl set-hostname WebServer-IP17
[root@centos79 </sub>]# exit
# 修改NTP服务器地址为阿里云的NTP   启用时钟同步服务
[root@webserver-ip17 <sub>]# timedatectl set-timezone Asia/Shanghai
[root@webserver-ip17 </sub>]#sed -i /^server/cserver ntp.aliyun.com iburst /etc/chrony.conf
[root@webserver-ip17 <sub>]# systemctl enable --now  chronyd.service
# 安装Apache
[root@webserver-ip17 </sub>]# yum -y install httpd
# 定义web主页文件
[root@webserver-ip17 <sub>]# hostname > /var/www/html/indexTmp.html
[root@webserver-ip17 </sub>]# hostname -I >> /var/www/html/indexTmp.html

# 将文件 /var/www/html/indexTmp.html 内两行文字合并成一行，便于后面测试观测
[root@webserver-ip17 <sub>]# cat /var/www/html/indexTmp.html | xargs > /var/www/html/index.html
[root@webserver-ip17 </sub>]# ll /var/www/html/                                                 
total 8
-rw-r--r-- 1 root root 30 Mar 31 23:07 index.html
-rw-r--r-- 1 root root 31 Mar 31 23:07 indexTmp.html
[root@webserver-ip17 <sub>]# rm -rf /var/www/html/indexTmp.html                                
[root@webserver-ip17 </sub>]# ll /var/www/html/                 
total 4
-rw-r--r-- 1 root root 30 Mar 31 23:07 index.html

# 启动Apache服务，并开机自启
[root@webserver-ip17 <sub>]# systemctl  enable --now httpd

# 验证
[root@webserver-ip17 </sub>]# curl 192.168.250.17                                               
webserver-ip17 192.168.250.17
[root@webserver-ip17 <sub>]# 

#####################################################################################
#### 同样的方式完成另外一台 webserver-ip27 192.168.250.27 的Apache的配置和调试
# 修改主机名、同步时间等基础环境配置需要按常规完成好
[root@centos79 </sub>]# hostnamectl set-hostname WebServer-IP17
[root@centos79 <sub>]# exit
# 修改NTP服务器地址为阿里云的NTP   启用时钟同步服务
[root@webserver-ip27 </sub>]# timedatectl set-timezone Asia/Shanghai
[root@webserver-ip27 <sub>]#sed -i /^server/cserver ntp.aliyun.com iburst /etc/chrony.conf
[root@webserver-ip27 </sub>]# systemctl enable --now  chronyd.service

# 安装Apache、配置主页、启动服务 一次搞定
[root@webserver-ip27 <sub>]# yum -y install httpd;hostname > /var/www/html/indexTmp.html;hostname -I >> /var/www/html/indexTmp.html;cat /var/www/html/indexTmp.html | xargs > /var/www/html/index.html;rm -rf /var/www/html/indexTmp.html;systemctl  enable --now httpd

# 验证
[root@webserver-ip27 </sub>]# curl 192.168.250.27
webserver-ip27 192.168.250.27
[root@webserver-ip27 ~]#

4.2 配置与LVS相关的配置

简要概述：因为我们在利用keepalived实现LVS-DR模式的WEB服务负载均衡，所以需要对后端两台服务器的ARP宣告、VIP绑定做好相应的配置，这样在LVS + keepalived 配置好后就可以直接按照LVS-DR模式访问后端的服务器了，因为这个过程在我的博客的前面文章已经详细介绍过，对这两台服务器的配置直接用脚本来完成了。

#### lvs_dr_rs.sh 内容,在VS-Code内修改好上传到两个WEB-RS服务器上运行

#!/bin/bash
vip=192.168.250.100
mask=255.255.255.255
dev=lo:1

case $1 in
start)
    echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
    ifconfig $dev $vip netmask $mask 
    echo "The RS Server is Ready!"
    ;;
stop)
    ifconfig $dev down
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
    echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
    echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
    echo "The RS Server is Canceled!"
    ;;
*) 
    echo "Usage: $(basename $0) start|stop"
    exit 1
    ;;
esac

## IP192.168.250.17  上的配置过程
[root@webserver-ip17 <sub>]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring lvs_dr_rs.sh...
  100%     728 bytes  728 bytes/sec 00:00:01       0 Errors  

[root@webserver-ip17 </sub>]# bash
[root@webserver-ip17 <sub>]# bash lvs_dr_rs.sh 
Usage: lvs_dr_rs.sh start|stop
[root@webserver-ip17 </sub>]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@webserver-ip17 <sub>]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 192.168.250.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a3:a8:67 brd ff:ff:ff:ff:ff:ff
    inet 192.168.250.17/24 brd 192.168.250.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea3:a867/64 scope link 
       valid_lft forever preferred_lft forever
[root@webserver-ip17 </sub>]# 

## IP192.168.250.27  上的配置过程
[root@webserver-ip27 <sub>]# rz
rz waiting to receive.
Starting zmodem transfer.  Press Ctrl+C to cancel.
Transferring lvs_dr_rs.sh...
  100%     728 bytes  728 bytes/sec 00:00:01       0 Errors  

[root@webserver-ip27 </sub>]# bash lvs_dr_rs.sh start
The RS Server is Ready!
[root@webserver-ip27 <sub>]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet 192.168.250.100/32 scope global lo:1
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a3:fb:92 brd ff:ff:ff:ff:ff:ff
    inet 192.168.250.27/24 brd 192.168.250.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea3:fb92/64 scope link 
       valid_lft forever preferred_lft forever
[root@webserver-ip27 </sub>]#

5. 配置keepalived服务器

第一步：先配置实现master/slave的 Keepalived 单主架构，并测试成功。第二步再配置和LVS的ipvs服务相关的，实现和后端WEB-RS的通信。

5.1 配置 Keepalived 实现master/slave的单主架构

#### IP 192.168.250.18 Keepalived-IP18 的 Keepalived 配置
[root@CentOS84-IP18 ]#hostnamectl set-hostname Keepalived-IP18
[root@CentOS84-IP18 ]#exit

[root@Keepalived-IP18 ]#timedatectl set-timezone Asia/Shanghai
[root@Keepalived-IP18 ]#sed -i /^server/cserver ntp.aliyun.com iburst /etc/chrony.conf
[root@Keepalived-IP18 ]#systemctl enable --now  chronyd.service
[root@Keepalived-IP18 ]#systemctl restart  chronyd.service            

[root@Keepalived-IP18 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a3:e8:6b brd ff:ff:ff:ff:ff:ff
    inet 192.168.250.18/24 brd 192.168.250.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea3:e86b/64 scope link 
       valid_lft forever preferred_lft forever
[root@Keepalived-IP18 ]#

[root@Keepalived-IP18 ]#dnf info keepalived
Last metadata expiration check: 19:41:03 ago on Wed 30 Mar 2022 10:03:06 PM CST.
Installed Packages
Name         : keepalived
Version      : 2.1.5
....................

[root@Keepalived-IP18 ]#dnf install keepalived -y
[root@Keepalived-IP18 ]#keepalived -v
Keepalived v2.1.5 (07/13,2020)

Copyright(C) 2001-2020 Alexandre Cassen, <acassen@gmail.com>

Built with kernel headers for Linux 4.18.0
Running on Linux 4.18.0-305.3.1.el8.x86_64 #1 SMP Tue Jun 1 16:14:33 UTC 2021
......................
# 备份默认的 keepalived.conf 配置文件并按照规划拓扑结构修改相应的配置文件。配置文件详细的每行含义请参见文章末尾的详细说明。
[root@Keepalived-IP18 ]#cp /etc/keepalived/keepalived.conf,.bak
[root@Keepalived-IP18 ]#vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
# 全局设置：邮箱设置，要结合系统的/etc/mail.rc 和通知脚本来实现故障通知。
global_defs 
   notification_email 
     root@shone.cn
   
   notification_email_from admin@shone.cn
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id KA-IP18
   vrrp_skip_check_adv_addr
   #vrrp_strict             # 严格模式建议关闭掉
   vrrp_garp_interval 0
   vrrp_gna_interval 0
   #vrrp_mcast_group4 234.0.0.66   #可以用自定义多播地址通告VRRP信息，建议用下面的单播



vrrp_instance VI_IP100 
    state MASTER
    interface eth0
    virtual_router_id 100
    priority 100
    advert_int 1
    authentication 
        auth_type PASS
        auth_pass shone888
    
    virtual_ipaddress 
        192.168.250.100 dev eth0 label eth0:1
    
# 下面是单播的配置  
    unicast_src_ip 192.168.250.18                            
    unicast_peer 
        192.168.250.28
    


# 全部配置好启动服务并设定开机自启
[root@Keepalived-IP28 ]#systemctl enable --now keepalived

#### IP 192.168.250.28 Keepalived-IP28 的 Keepalived 配置

[root@CentOS84-IP28 ]#hostnamectl set-hostname Keepalived-IP28
[root@CentOS84-IP28 ]#exit

[root@Keepalived-IP28 ]#timedatectl set-timezone Asia/Shanghai
[root@Keepalived-IP28 ]#sed -i /^server/cserver ntp.aliyun.com iburst /etc/chrony.conf
[root@Keepalived-IP28 ]#systemctl enable --now  chronyd.service
[root@Keepalived-IP28 ]#systemctl restart  chronyd.service            

[root@Keepalived-IP28 ]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a3:e2:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.250.28/24 brd 192.168.250.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea3:e2bf/64 scope link 
       valid_lft forever preferred_lft forever
[root@Keepalived-IP28 ]#

[root@Keepalived-IP28 ]#dnf info keepalived
Last metadata expiration check: 19:41:03 ago on Wed 30 Mar 2022 10:03:06 PM CST.
Installed Packages
Name         : keepalived
Version      : 2.1.5
....................

[root@Keepalived-IP28 ]#dnf install keepalived -y
[root@Keepalived-IP28 ]#keepalived -v
Keepalived v2.1.5 (07/13,2020)
......................

# 备份默认的 keepalived.conf 配置文件并按照规划拓扑结构修改相应的配置文件。配置文件详细的每行含义请参见文章末尾的详细说明。
[root@Keepalived-IP28 ]#cp /etc/keepalived/keepalived.conf,.bak
[root@Keepalived-IP28 ]#vim /etc/keepalived/keepalived.conf
! Configuration File keepalived实现单主的LVS-DR模式
 keepalived用脚本实现切换
 实现master/slave的Keepalived单主架构
 Linux-利用keepalived实现lvs的高可用性
 LVS+Keepalived+Nginx+Tomcat高可用集群搭建(转)
 LVS+Keepalived 高可用群集实战部署

Keepalived+LVS实战案例： 单主架构实现WEB负载均衡及可用