Docker+Nginx+Keepalived实现架构高可用

Posted 2020-12-04

一、背景

通过keepalived实现nginx高可用，由于在家不想弄多台主机来搞，所以将运行环境用docker封装来模拟跨主机

docker基础镜像：centos

 

说之前，简单介绍一下：

Keepalived是基于vrrp协议的一款高可用软件。Keepailived有一台主服务器和多台备份服务器，在主服务器和备份服务器上面部署相同的服务配置，使用一个虚拟IP地址对外提供服务，当主服务器出现故障时，虚拟IP地址会自动漂移到备份服务器。

 

双机高可用方法目前分为两种：

• 双机主从（也叫双机热备）
• 双机主主 （也叫双机互备）

下述介绍，高可用中的双机主从模式，双机主主模式，主要是keepalived.conf配置会有所不同，方法都是一样。

二、具体操作

1、安装centos 镜像

docker pull centos

说明：通过用centos镜像来安装高可用所需要的所有环境，再启两个容器，再真实模拟跨主机的场景

 

2、在centos上安装所需环境（nginx和其它工具）

运行centos容器

docker run -it centos /bash/bin

 

安装依赖和所需要的包　　

#使用yum安装nginx需要包括Nginx的库，安装Nginx的库

rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm

# 使用下面命令安装nginx
#yum install nginx

#安装网络包(需要使用ifconfig和ping命令)
yum install net-tools

#安装vim
yum install vim

 

3、在centos安装keepalvied

#安装keepalived环境依赖

yum install -y gcc openssl-devel popt-devel

#安装keepalived

通过yum install keepalived



#或者通过源码安装

wget http://124.205.69.132/files/90630000053A2BB4/www.keepalived.org/software/keepalived-1.3.4.tar.gz

tar zxvf keepalived-1.3.4.tar.gz 

cd keepalived-1.3.4

./configure --prefix=/usr/local/keepalived

make && make install



拷贝几个文件到CentOS7环境中：

cp keepalived-1.3.4/keepalived/etc/init.d/keepalived /etc/init.d/
mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp keepalived-1.3.4/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

 

4、修改/etc/keepalived/keepalived.conf文件

! Configuration File for keepalived
global_defs {
    notification_email {
    	[e-mail]
    }
    notification_email_from [e-mail]
    smtp_server mail.example.com
    smtp_connect_timeout 30
    router_id LVS_DEVEL  
}


vrrp_script chk_nginx {
    script "/etc/keepalived/nginx_check.sh"  
    interval 2
    weight -5
    fall 3
    rise 2
}


vrrp_instance VI_1 {
    state MASTER   
    interface eth0   
    virtual_router_id 2  
    priority 101  
    advert_int 2
    authentication {
        auth_type PASS  
        auth_pass 1111
    }
    virtual_ipaddress {
        172.17.0.210   
    }
    track_script {
       chk_nginx 
    }

}

　　

/etc/keepalived/check_nginx.sh文件

A=`ps -ef | grep nginx | grep -v grep | wc -l`
if [ $A -eq 0 ];then
    nginx
    sleep 2
    if [ `ps -ef | grep nginx | grep -v grep | wc -l` -eq 0 ];then
        #killall keepalived
        ps -ef|grep keepalived|grep -v grep|awk \'{print $2}\'|xargs kill -9 
    fi

fi　　

 

再对check_nginx.sh赋于执行权限：

chmod +x check_nginx.sh

 

注：keepalived是通过检测keepalived进程是否存在判断服务器是否宕机，如果keepalived进程在但是nginx进程不在了那么keepalived是不会做主备切换，所以我们需要写个脚本来监控nginx进程是否存在，如果nginx不存在就将keepalived进程杀掉。

在主nginx上需要编写nginx进程检测脚本（check_nginx.sh），判断nginx进程是否存在，如果nginx不存在就将keepalived进程杀掉，并将vip漂移到备份机器上

 

5、设置开机启动

chkconfig keepalived on

或者

systemctl enable keepalived.service  设置开机自动启动

启动keepalived服务：

systemctl start keepalived.service 启动

 

6、安装所有需要的依赖和环境后，将容器新增的内容重新提交

docker commit 5d112 centos_keepalived_nginx:v1

注：5d112为，上述安装软件所对应的容器id

 

6、启动含有(keepalived+nginx)的容器

docker run --privileged  -tid --name  keepalived_master centos_keepalived_nginx:v1 /usr/sbin/init

 

 

进入keepalived_master容器：

docker exec -it keepalived_master bash

进入/usr/share/nginx/html，修改index.html文件

修改标题为:

Welcome to nginx Master!

7、启动keepalived_salve容器

#启动一个容器

docker run --privileged  -tid --name  keepalived_slave centos_keepalived_nginx:v1 /usr/sbin/init

#进入容器

docker exec -it keepalived_slave bash

 

8、修改keepalived_salve容器中nginx index.html文件

vim /usr/share/nginx/html/index.html

 

 

修改标题为:

Welcome to nginx Slave!

 

9、修改keepalived_salve容器中keepalived.conf文件 (master容器中，保持和镜像中设置一样即可，不需要更改)

! Configuration File for keepalived
global_defs {
notification_email {
[e-mail]
}
notification_email_from [e-mail]
smtp_server mail.example.com
smtp_connect_timeout 30
router_id LVS_DEVEL
}

vrrp_script chk_nginx {
script "/etc/keepalived/nginx_check.sh"
interval 2
weight -5
fall 3
rise 2
}


vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 2
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.17.0.210
}
track_script {
chk_nginx
}

}

其实，由配置中可以看出，主要是state和priority两个参数的调整，其中master节点的priority值一定要比backup大才行！

 

原理说明：
1、 通过vrrp协议广播，每个keepalived vrrp都去争取master
2、 以virtual_router_id为组队标识。  同为一个vip服务的keepalived的virtual_router_id要保持相同

3、 以priority 为权值，同一个virtual_router_id下那个priority大那个就是master,其它为backup

 

 

改完之后，重新加载

systemctl daemon-reload 
systemctl restart keepalived.service

 

10、验证

查看两个容器中keepalived服务状态

systemctl status keepalived.service 

keepalived_master服务状态效果：

keepalived_slave服务状态效果图：

 

可以看到，keepalived服务运行正常

 

启动nginx: nginx

 

在master容器中 curl 172.17.0.210

 

在slave容器中 curl 172.17.0.210:

可以看现，此时master和slave容器两边通过虚拟vip : 172.17.0.210 访问nginx数据，请求返回的数据都是master容器中nginx配置的数据： welcome to nginx master

 

继续验证，关掉master容器的keepalived服务：

 

 验证得到的结果是当master容器中的keepalived服务关掉后，curl 172.17.0.210请求返回的数据来自slave，welcome to nginx slave

 

再继续验证，把关掉master容器的keepalived服务再开启：

 

可以看到，当master容器中的keepalived服务开启后，请求返回的数据会再次转到master中。

 

到此，所有的验证和预期的一致，也达到我们借助docker为基础来实现了整套基于Nginx+Keepalived高可用的方案了。

 

三、Keepalived服务命令

• systemctl daemon-reload  重新加载
• systemctl enable keepalived.service  设置开机自动启动
• systemctl disable keepalived.service 取消开机自动启动
• systemctl start keepalived.service 启动
• systemctl stop keepalived.service停止
• systemctl status keepalived.service  查看服务状态