Keepalived+LVS实现LNMP网站的高可用部署
Posted
tags:
篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Keepalived+LVS实现LNMP网站的高可用部署相关的知识,希望对你有一定的参考价值。
项目需求 当我们访问某个网站的时候可以在浏览器中输入IP或者域名链接到Web Server进行访问,如果这个Web Server挂了,那么整个系统都无法使用,用户也就不能进行正常的访问,这种情况将对公司产生一定的影响。这就是我们常说的系统中的单点故障。这部分的单点故障可以通过引入负载均衡器和至少另一个Web Server来缓解。同时由于有多台服务器同时提供服务,也加大了系统的负载能力提高了性能。
因此我们采用LVS的负载均衡技术,将前端请求按照设定规则调度到后端服务器,并与keepalived相结合实现高可用负载均衡。
项目拓扑
项目环境
主机名 | 主机IP | 主机角色 |
---|---|---|
K1 | 192.168.36.110 | Keepalived-Master |
K2 | 192.168.36.111 | Keepalived-Backup |
WEB1 | 192.168.36.112 | nginx、php |
WEB2 | 192.168.36.113 | Nginx、PHP |
NFS | 192.168.36.114 | NFS |
Mariadb-M | 192.168.36.115 | Mariadb-Master |
Mariadb-S | 192.168.36.116 | Mariadb-Slave |
开始部署:Keepalived服务器配置
安装Keepalived
[[email protected] ~]#yum install -y keepalived
修改Keepalived配置文件,开启邮件通知功能
[[email protected] ~]#vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs # 全局配置
notification_email
[email protected] # keepalived 发生故障切换时邮件发送的对象,可以按行区分写多个
notification_email_from [email protected] # 设置邮件发送地址
smtp_server 127.0.0.1 # smtp服务器地址
smtp_connect_timeout 30 # 指定smtp连接超时时间
router_id K1.mylinuxops.com # 运行keepalived服务器标识,发送邮件时显示在邮件标题中的信息
vrrp_skip_check_adv_addr # 所有报文都检查比较消耗性能,此配置为如果收到的报文和上一个报文是同一个路由器则跳过检查报文中的源地址
#vrrp_strict # 严格遵守VRRP协议,不允许状况:1,没有VIP地址,2.单播邻居,3.在VRRP版本2中有IPv6地址
vrrp_iptables # 严格遵守VRRP防火墙规则
vrrp_garp_interval 0 # ARP保温发送延迟
vrrp_gna_interval 0 # 消息发送延迟
vrrp_instance VI_1 # vrrp实例定义
state MASTER # 在此虚拟路由器上节点的初始状态:其中所有服务器里只有一个可以是MASTER节点,其余的是BACKUP节点
interface ens33 # 指定HA检测网络的接口,即网卡名称
virtual_router_id 27 # 当前虚拟路由器的惟一标识,范围是0-255
priority 100 # 当前主机在此虚拟路径器中的优先级;范围1-254。主服务器一定要高于备用服务器,且两者之间的数值差越小越好
advert_int 1 # vrrp通告间隔
authentication # 存储的验证类型和密码以进行验证
auth_type PASS # 进行验证类型:类型仅可以设置成PASS和AH两种
auth_pass 1111 # 进行验证的密码:在同一个vrrp_instance中,使用相同的密码才能进行正确的通信
unicast_src_ip 192.168.36.110 # 单播配置
unicast_peer
192.168.36.111 # 目标主机IP
virtual_ipaddress # 虚拟IP的网络地址,即VIP地址
192.168.36.100 dev ens33 label ens33:0
192.168.36.200 dev ens33 label ens33:1
# 定义邮件通知脚本
notify_master "/etc/keepalived/notify.sh master" # 当前节点成为主节点时触发的脚本
notify_backup "/etc/keepalived/notify.sh backup" # 当前节点转为备节点时触发的脚本
notify_fault "/etc/keepalived/notify.sh fault" # 当前节点转为“失败”状态时触发的脚本
# 编写邮件通知脚本
[[email protected] ~]#vim /etc/keepalived/notify.sh
#!/bin/bash
contact=‘[email protected]‘ # 通知的邮箱(首先需要确保能连通外网,否则通知不过去)
notify()
mailsubject="$(hostname) to be $1, vip转移" # 主题
mailbody="$(date +‘%F %T‘): vrrp transition, $(hostname) changed to be $1" # 邮件内容
echo "$mailbody" | mail -s "$mailsubject" $contact # 邮件发送的内容
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) master|backup|fault"
exit 1
;;
esac
# 添加执行权限
[[email protected] ~]#chmod a+x /etc/keepalived/notify.sh
# 邮箱配置
[[email protected] ~]#yum install -y mailx
[[email protected] ~]#vim /etc/mail.rc
set bsdcompat
set [email protected] # 接收邮件的邮箱
set smtp=smtp.qq.com
set [email protected]
set smtp-auth-password=kosulaxbbhxrgaci # 邮箱授权码(我的邮箱-->设置-->账户-->开启POP3/SMTP等服务,生成授权码)
[[email protected] ~]#chmod a+x /etc/mail.rc # 文件添加执行权限
# 重启keepalived服务
[[email protected] ~]#systemctl restart keepalived
# 生成VIP地址(注:Master存活时,VIP在Master上,Slave上不会存在VIP。当Master宕机,VIP将调到Slave上)
[[email protected] ~]#ifconfig
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.36.100 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:56:39:e8 txqueuelen 1000 (Ethernet)
ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.36.200 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:56:39:e8 txqueuelen 1000 (Ethernet)
# 将keepalived配置文件 scp 到BACKUP服务器中
[[email protected] ~]#scp /etc/keepalived/keepalived.conf 192.168.36.104:/etc/keepalived/keepalived.conf
[email protected]‘s password:
keepalived.conf 100% 1374 1.4MB/s 00:00
# BACKUP服务器配置,其余配置与MASTER相同
[[email protected] ~]#vim /etc/keepalived/keepalived.conf
....
state BACKUP # 修改为BACKUP节点
interface ens33
virtual_router_id 37 # 修改标识为37
priority 90 # 修改优先级,需要比 MASTER 节点低
advert_int 1
authentication
auth_type PASS
auth_pass 1111
unicast_src_ip 192.168.36.111
unicast_peer
192.168.36.110
....
# 编写邮件通知脚本
[[email protected] ~]#vim /etc/keepalived/notify.sh
#!/bin/bash
contact=‘[email protected]‘ # 通知的邮箱(首先需要确保能连通外网,否则通知不过去)
notify()
mailsubject="$(hostname) to be $1, vip转移" # 主题
mailbody="$(date +‘%F %T‘): vrrp transition, $(hostname) changed to be $1" # 邮件内容
echo "$mailbody" | mail -s "$mailsubject" $contact # 邮件发送的内容
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) master|backup|fault"
exit 1
;;
esac
# 添加执行权限
[[email protected] ~]#chmod a+x /etc/keepalived/notify.sh
# 邮箱配置
[[email protected] ~]#yum install -y mailx
[[email protected] ~]#vim /etc/mail.rc
set bsdcompat
set [email protected] # 接收邮件的邮箱
set smtp=smtp.qq.com
set [email protected]
set smtp-auth-password=kosulaxbbhxrgaci # 邮箱授权码(我的邮箱-->设置-->账户-->开启POP3/SMTP等服务,生成授权码)
[[email protected] ~]#chmod a+x /etc/mail.rc # 文件添加执行权限
# 重启keepalived服务,并进行宕机测验,查看VIP跳转到K2服务器上
[[email protected] ~]#ifconfig
ens33:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.36.100 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:56:39:e8 txqueuelen 1000 (Ethernet)
ens33:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.36.200 netmask 255.255.255.255 broadcast 0.0.0.0
ether 00:0c:29:56:39:e8 txqueuelen 1000 (Ethernet)
VIP跳转邮箱接收邮件
搭建WEB站点(两个WEB站点执行相同操作)
# 编写Nginx编译安装脚本
[[email protected] ~]#vim nginx.sh
#!/bin/bash
yum install -y vim lrzsz tree screen psmisc lsof tcpdump wget ntpdate gcc gcc-c++ glibc glibc-devel pcre pcre-devel openssl openssl-devel systemd-devel net-tools iotop bc zip unzip zlib-devel bash-completion nfs-utils automake libxml2 libxml2-devel libxslt libxslt-devel perl perl-ExtUtils-Embed &>/dev/null
wget https://nginx.org/download/nginx-1.14.2.tar.gz &>/dev/null
cd nginx-1.14.2/
./configure --prefix=/apps/nginx --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre --with-stream --with-stream_ssl_module --with-stream_realip_module &>/dev/null
make && make install
useradd nginx -s /sbin/nologin -u 2000
chown nginx.nginx -R /apps/nginx/
echo "\n=======================版本==================================\n"
/apps/nginx/sbin/nginx -V
# 给脚本添加执行权限
[[email protected] ~]#chmod a+x nginx.sh
# 启动安装脚本
[[email protected] ~]#./nginx.sh
# 成功安装Nginx,做nginx命令软链接
[[email protected] ~]#ln -sv /apps/nginx/sbin/nginx /usr/sbin/
‘/usr/sbin/nginx’ -> ‘/apps/nginx/sbin/nginx’
# 启动Nginx
[[email protected] ~]#nginx
# 查看80端口
[[email protected] ~]#ss -ntl
# 修改Nginx配置文件,使其开启php页面访问功能
[[email protected] ~]#vim /apps/nginx/conf/nginx.conf
2 user nginx nginx;
9 pid logs/nginx.pid;
39 charset utf-8;
42 location /
43 root html;
44 index index.php index.html index.htm;
45
64 location ~ \.php$
65 root /apps/nginx/html;
66 fastcgi_pass 127.0.0.1:9000;
67 fastcgi_index index.php;
68 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
69 include fastcgi_params;
70
# 编写php状态页面
[[email protected] ~]#vim /apps/nginx/html/index.php
<?php
phpinfo();
?>
# Nginx启动前对配置文件进行检查
[[email protected] ~]#nginx -t
# 重新加载Nginx配置文件
[[email protected] ~]#nginx -s reload
nginx: the configuration file /apps/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /apps/nginx/conf/nginx.conf test is successful
# 安装php-fpm模块
[[email protected] ~]#yum install -y php-fpm php-mysql
# 编写php-fpm模块配置文件
[[email protected] ~]#vim /etc/php-fpm.d/www.conf
12 listen = 127.0.0.1:9000
33 listen.mode = 0666
39 user = nginx
41 group = nginx
# 启动php-fpm
[[email protected] ~]#systemctl restart php-fpm
# 9000端口查看
[[email protected] ~]#ss -ntl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 127.0.0.1:9000 *:*
两个keepalived服务器中添加 web 的虚拟服务器
[[email protected] ~]#vim /etc/keepalived/keepalived.conf
....
virtual_server 192.168.36.100 80 # 虚拟服务器的虚拟IP地址和服务的端口号
delay_loop 6 # 系统执行健康检查的时间间隔
lb_algo wrr # lvs调度的算法:wrr轮询算法
lb_kind DR # LVS的DR直接路由机制
protocol TCP # 指定转发协议,TCP/UDP
real_server 192.168.36.112 80 # 实际服务器IP地址和端口号
weight 1 # 权重值
TCP_CHECK # 通过tcpcheck判断RealServer的健康状态
connect_port 80 # 检测连接端口
connect_timeout 5 # 连接超时时间
nb_get_retry 3 # 重连次数
delay_before_retry 3 # 重连时间间隔
real_server 192.168.36.113 80
weight 1
TCP_CHECK
connect_port 80
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
# 重新启动keepalived服务
[[email protected] ~]#systemctl restart keepalived
# 查看生成的ipvsadm规则
[[email protected] ~]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.36.100:80 wrr
-> 192.168.36.112:80 Route 1 0 0
-> 192.168.36.113:80 Route 1 0 0
两个WEB服务器创建lvs检测脚本(步骤相同)
[[email protected] ~]#vim lvs_dr_rs.sh
#!/bin/bash
vip=192.168.36.100
mask=‘255.255.255.255‘
dev=lo:1
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
# 添加执行权限并运行脚本
[[email protected] ~]#chmod a+x ./lvs_dr_rs.sh
[[email protected] ~]#./lvs_dr_rs.sh start
The RS Server is Ready!
# 生成检测的虚拟IP
[[email protected] ~]#ifconfig lo:1
lo:1: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.36.100 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
状态页面查看
VIP状态页
搭建数据库主从复制服务器
Master服务器
# yum安装mariadb服务
[[email protected] ~]#yum install -y mariadb-server
# 启动数据库服务
[[email protected] ~]#systemctl restart mariadb
# 修改mariadb配置文件
[[email protected] ~]#vim /etc/my.cnf
[mysqld]
server_id=1 # ID号
binlog_format=row # 基于行复制
log-bin=/data/bin/mysql-bin # 生成二进制文件的目录与格式
# 创建二进制文件存放的目录
[[email protected] ~]#mkdir /data/bin
# 授予目录所属关系
[[email protected] ~]#chown mysql.mysql /data/bin/ -R
# 重新启动mariadb服务
[[email protected] ~]#systemctl restart mariadb
# 执行安全脚本
[[email protected] ~]#mysql_secure_installation
# 进入数据库
[[email protected] ~]#mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 19
Server version: 5.5.60-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
MariaDB [(none)]> grant replication slave on *.* to [email protected]‘192.168.36.%‘ identified by ‘centos‘; # 添加slave复制权限
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> flush privileges; # 刷新权限
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> reset master; # 重置master二进制文件大小
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show master logs; # 查看并记录
+------------------+-----------+
| Log_name | File_size |
+------------------+-----------+
| mysql-bin.000001 | 245 |
+------------------+-----------+
1 row in set (0.00 sec)
Slave服务器
# yum安装mariadb服务
[[email protected] ~]#yum install -y mariadb-server
# 启动数据库服务
[[email protected] ~]#systemctl restart mariadb
# 修改mariadb配置文件
[[email protected] ~]#vim /etc/my.cnf
[mysqld]
server_id=2 # ID号
read_only # 只读
# 重新启动mariadb服务
[[email protected] ~]#systemctl restart mariadb
# 执行安全脚本
[[email protected] ~]#mysql_secure_installation
# 进入数据库
[[email protected] ~]#mysql -uroot -p123456
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 15
Server version: 5.5.60-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type ‘help;‘ or ‘\h‘ for help. Type ‘\c‘ to clear the current input statement.
MariaDB [(none)]> CHANGE MASTER TO # Slave节点添加同步Master数据库的语句
-> MASTER_HOST=‘192.168.36.115‘,
-> MASTER_USER=‘repluser‘,
-> MASTER_PASSWORD=‘centos‘,
-> MASTER_PORT=3306,
-> MASTER_LOG_FILE=‘mysql-bin.000001‘,
-> MASTER_LOG_POS=245;
Query OK, 0 rows affected (0.01 sec)
MariaDB [(none)]> slave start; # 启用从节点
Query OK, 0 rows affected (0.00 sec)
MariaDB [(none)]> show slave status\G; # 状态查看
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.36.115
Master_User: repluser
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 245
Relay_Log_File: mariadb-relay-bin.000002
Relay_Log_Pos: 529
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes # IO、SQL线程已经启动,数据同步
Slave_SQL_Running: Yes
Master、Slave数据同步测试
MariaDB [(none)]> create database darius; # Master节点创建一个darius数据库
Query OK, 1 row affected (0.00 sec)
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| darius |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
MariaDB [(none)]> show databases; # Slave查看,如同步成功,则主从复制创建完成。
+--------------------+
| Database |
+--------------------+
| information_schema |
| darius |
| mysql |
| performance_schema |
+--------------------+
4 rows in set (0.00 sec)
两个keepalived服务器添加mariad虚拟服务器
[[email protected] ~]#vim /etc/keepalived/keepalived.conf
....
virtual_server 192.168.36.200 3306
delay_loop 6
lb_algo wrr
lb_kind DR
protocol TCP
real_server 192.168.36.115 3306
weight 1
TCP_CHECK
connect_port 3306
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
real_server 192.168.36.116 3306
weight 1
TCP_CHECK
connect_port 3306
connect_timeout 5
nb_get_retry 3
delay_before_retry 3
....
# 生成ipvsadm规则
[[email protected] ~]#ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.36.100:80 wrr
-> 192.168.36.112:80 Route 1 0 0
-> 192.168.36.113:80 Route 1 0 0
TCP 192.168.36.200:3306 wrr
-> 192.168.36.115:3306 Route 1 0 0
-> 192.168.36.116:3306 Route 1 0 0
两个mariadb服务器创建lvs检测脚本(步骤相同)
[[email protected] ~]#vim lvs_dr_rs.sh
#!/bin/bash
vip=192.168.36.200
mask=‘255.255.255.255‘
dev=lo:1
case $1 in
start)
echo 1 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 1 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 2 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 2 > /proc/sys/net/ipv4/conf/lo/arp_announce
ifconfig $dev $vip netmask $mask #broadcast $vip up
#route add -host $vip dev $dev
echo "The RS Server is Ready!"
;;
stop)
ifconfig $dev down
echo 0 > /proc/sys/net/ipv4/conf/all/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_ignore
echo 0 > /proc/sys/net/ipv4/conf/all/arp_announce
echo 0 > /proc/sys/net/ipv4/conf/lo/arp_announce
echo "The RS Server is Canceled!"
;;
*)
echo "Usage: $(basename $0) start|stop"
exit 1
;;
esac
# 给脚本添加执行权限
[[email protected] ~]#chmod a+x lvs_dr_rs.sh
# 运行脚本
[[email protected] ~]#./lvs_dr_rs.sh start
The RS Server is Ready!
# 生成检测的虚拟IP
[[email protected] ~]#ifconfig lo:1
lo:1: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 192.168.36.200 netmask 255.255.255.255
loop txqueuelen 1000 (Local Loopback)
搭建NFS服务器
# 修改NFS配置文件,设置将要共享的目录
[[email protected] ~]#vim /etc/exports
/data *(rw,no_root_squash)
# 重新启动NFS服务,并设置开机启动
[[email protected] ~]#systemctl restart nfs
[[email protected] ~]#systemctl enable nfs
# 查看NFS共享目录
[[email protected] ~]#exportfs -v
/data <world>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
# 解压wordpress包
[[email protected] ~]#unzip wordpress-5.0-zh_CN.zip
# 将wordpress包内文件移动到共享目录中,进行共享
[[email protected] ~]#mv wordpress/* /data/
[[email protected] ~]#cd /data/
# 生成wordpress配置文件
[[email protected] data]#mv wp-config-sample.php wp-config.php
# 修改wordpress配置文件
[[email protected] data]#vim wp-config.php
...
/** WordPress数据库的名称 */
define(‘DB_NAME‘, ‘wordpress‘);
/** MySQL数据库用户名 */
define(‘DB_USER‘, ‘wpuser‘);
/** MySQL数据库密码 */
define(‘DB_PASSWORD‘, ‘centos‘);
/** MySQL主机 */
define(‘DB_HOST‘, ‘192.168.36.200‘);
/** 创建数据表时默认的文字编码 */
define(‘DB_CHARSET‘, ‘utf8‘);
...
将NFS共享的文件挂载到两个WEB服务器中
# 写入fstab文件中,开机自动挂载
[[email protected] ~]#vim /etc/fstab
192.168.36.114:/data /apps/nginx/html nfs _netdev,defaults 0 0
# 查看挂载情况
[[email protected] ~]#df -h
Filesystem Size Used Avail Use% Mounted on
192.168.36.114:/data 95G 3.9G 92G 5% /apps/nginx/html
访问测试
多次宕机实验依旧能访问,起到web服务的高可用功能。
以上是关于Keepalived+LVS实现LNMP网站的高可用部署的主要内容,如果未能解决你的问题,请参考以下文章