Nginx+Keepalived搭建高可用负载平衡WEB 集群

Posted

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了Nginx+Keepalived搭建高可用负载平衡WEB 集群相关的知识，希望对你有一定的参考价值。

nginx+Keepalived搭建高可用负载平衡WEB 集群

 

1.1环境规划：

技术分享 

Nginx_master:192.168.5.129

Nginx_backup:192.168.5.131

Tomcat:192.168.5.132 端口:8080,9080

 

操作系统：CentOS6.5 x86_64

内核版本：2.6.32-696.el6.x86_64

Nginx版本：nginx/1.12.0 nginx-1.12.0

Keepalived版本：Keepalived v1.2.13

    前端双Nginx+Keepalived，Nginx反向代理到后端tomcat集群实现负载均衡，Keepalived实现集群高可用，master_nginx故障后虚拟IP自动切换到backup_nginx。

    主nginx：192.168.5.129

    备nginx：192.168.5.131

    虚拟IP：192.168.5.200

    后端tomcat集群：192.168.5.132

    后端每个主机都开启两个端口提供业务：8080,9080

 

1.2 Keepalived介绍

Keepalived是一种用C编写的路由软件。该项目的主要目标是为Linux系统和基于Linux的基础设施提供简单而强大的负载平衡和高可用性设施。 负载平衡框架依赖于众所周知的广泛使用的Linux虚拟服务器（IPVS）内核模块，提供Layer4负载平衡。Keepalived实现了一组检查器，以动态和自适应地维护和管理负载平衡的服务器池，以保证其健康。另一方面，VRRP实现了高可用性 协议。VRRP是路由器故障切换的基础。此外，Keepalived实现了一组钩子到VRRP有限状态机，提供低级和高速协议交互。Keepalived框架可以单独使用或全部使用，以提供有弹性的基础设施。

2.1配置：Nginx+Tomcat实现反向代理

(1)下载安装jdk

[[email protected] src]# tar -xf jdk-8u111-linux-x64.gz

[[email protected]]# ln -s /usr/local/src/jdk

(2)设置环境变量JAVA_HOME:

export JAVA_HOME=/usr/local/jbk

export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

export CLASSSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib

[[email protected] src]#source /etc/profile

(3)测试看是否安装成功：

[[email protected] src]# java -version

java version "1.8.0_111"

Java(TM) SE Runtime Environment (build 1.8.0_111-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

(4)安装tomcat:

[[email protected] src]# tar zxvf apache-tomcat-8.5.9.tar.gz

[[email protected] src]# ln -s /usr/local/src/apache-tomcat-8.5.9 /usr/local/tomcat

[[email protected] src]# chown -R root.root /usr/local/tomcat/  

设置环境变量：

[[email protected] src]# tail -4 /etc/profile

export JAVA_HOME=/usr/local/jdk

export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

export CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$JAVA_HOME/lib/tools.jar

export TOMCAT_HOME=/usr/local/tomcat/

(5)配置两个tomcat的配置文/usr/local/tomcat/conf/server.xml

[[email protected]]# vim server.xml（第一个tomcat）

………………………………………………

 <Connector port="9080" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

    <!-- A "Connector" using the shared thread pool-->

    <!--

    <Connector executor="tomcatThreadPool"

               port="9080" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

………………………………………………

 <Host name="192.168.5.132"  appBase="/tomcat/helloworld"

            unpackWARs="true" autoDeploy="true">

<Context path="" docBase="" debug="0" reloadable="false" crossContext="true"/>（加上一句作用上面直接绑定的时候，下面这个必须要有,docBase="" 不然不得）

 

 

[[email protected] src]# cd /usr/local/tomcat2/conf （配置第二个tomcat配置文件）

[[email protected] conf]# vim server.xml（第一个tomcat）

………………………………………………

 <Connector port="8080" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

    <!-- A "Connector" using the shared thread pool-->

    <!--

    <Connector executor="tomcatThreadPool"

               port="8080" protocol="HTTP/1.1"

               connectionTimeout="20000"

               redirectPort="8443" />

………………………………………………

 <Host name="192.168.5.132"  appBase="/tomcat/helloworld2"

            unpackWARs="true" autoDeploy="true">

<Context path="" docBase="" debug="0" reloadable="false" crossContext="true"/>

 

(6)配置两个inde.jsp目录

[[email protected] ~]# cd /tomcat/

[[email protected] tomcat]# ls

helloworld  helloworld2

[[email protected] tomcat]# cd helloworld

[[email protected] helloworld]# ls

index.jsp

[[email protected] helloworld]# cat index.jsp

<html>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!</h1>

 

</BODY>

</HTML>

 

 

 

[[email protected] helloworld]# cd /tomcat/helloworld2/

[[email protected] helloworld2]# cat index.jsp

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!--------2</h1>

 

</BODY>

</HTML>

 

[[email protected] helloworld2]# cd /usr/local/tomcat/bin/

[[email protected] bin]# ./catalina.sh start    （启动tomcat）

Using CATALINA_BASE:   /usr/local/tomcat

Using CATALINA_HOME:   /usr/local/tomcat

Using CATALINA_TMPDIR: /usr/local/tomcat/temp

Using JRE_HOME:        /usr/local/jbk

Using CLASSPATH:       /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar

Tomcat started.

 

(7)测试tomcat：

[[email protected] helloworld2]# curl 192.168.5.132:8080

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!</h1>

 

</BODY>

</HTML>

 

[[email protected] helloworld2]# curl 192.168.5.132:9080

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!--------2</h1>

 

</BODY>

</HTML>

[[email protected] helloworld2]#

2.2配置nginx的反向代理

(1)配置nginx配置文件：

 

[[email protected] ~]# vim /usr/local/nginx/conf/nginx.conf

upstream mysite {

server 192.168.5.132:9080 max_fails=3 fail_timeout=20s weight=2;（添加tomcat作为后端服务）

server 192.168.5.132:8080 max_fails=3 fail_timeout=20s weight=1;

}

 

    server {

        listen       80;

        server_name  192.168.5.129;

 

        #charset koi8-r;

 

        #access_log  logs/host.access.log  main;

 

        location / {

           # root   /web;

           # index  index.jsp index.php index.html index.htm;

        proxy_pass http://mysite;

proxy_set_header Host  192.168.5.132;

                proxy_set_header X-Real-IP $remote_addr;     

                proxy_set_header X-Forwared-For $proxy_add_x_forwarded_for;

                proxy_set_header X-Forwared-For $proxy_add_x_forwarded_for;

}

（意思是访问这个IP的时候都是反向代理到后端的这些主机）

location /status {

    stub_status;

}

(2)重启nginx服务：

[[email protected] ~]# /usr/local/nginx/sbin/nginx -t

nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok

nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

[[email protected] ~]# /usr/local/nginx/sbin/nginx -s reload

 

(3)测试访问nginx就能访问到后端tomcat服务：

[[email protected] ~]# curl 192.168.5.129

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!--------2</h1>

 

</BODY>

</HTML>

 

[[email protected] ~]# curl 192.168.5.129

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!</h1>

 

</BODY>

</HTML>

 

[[email protected] ~]#

 

3.keepalived实现nginx高可用

3.1 安装

[[email protected] ~]# yum install -y keepalived

[[email protected] ~]# service keepalived start

[[email protected] ~]# service keepalived status

keepalived (pid  13083) 正在运行...

3.2 nginx监控脚本

[[email protected] ~]# cd /script/

[[email protected] script]# cat sc.sh

#!/bin/bash

 

pgrep nginx > check.log

if [ $? -ne 0 ] ;then    （当执行pgrep nginx时输出$?为0的时候证明nginx已经down掉，就把keepalived stop掉）

 service keepalived stop >> /script/check.log

 

fi

 

 

3.3  配置选项说明keepalived.conf

主机机器配置

[[email protected] ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

   notification_email {

     [email protected]      #接收警报的email地址,可以添加多个

   }

   notification_email_from [email protected]     ###发件人地址  

   smtp_server 127.0.0.1      #发送邮件的服务器

   smtp_connect_timeout 30###超时时间

   router_id LVS_DEVEL##load balancer 的标识 ID,用于email警报

}

vrrp_script chk_nginx {

    script "/script/sc.sh"####检测nginx状态的脚本路径

    interval 2

    weight -10

    fall 2  

    rise 2

}

vrrp_instance VI_1 {

    state MASTER ############ 辅机为 BACKUP  

    interface eth0####HA 监测网络接口  此项默认为eth0,因为本人在虚拟机下做测试所以将其改为eth2，可以用ifconfing  

    virtual_router_id 51#主、备机的 virtual_router_id 必须相同

    mcast_src_ip 192.168.100.128 ###本机IP地址

    priority 100########### 权值要比 back 高

    advert_int 1 #主备之间的通告间隔秒数

    authentication {

        auth_type PASS###默认配置 主备切换时的验证

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.5.200/24####虚拟ip，vip的地址  

    }

    track_script {

       chk_nginx### 执行监控的服务  

    }

}

Backup机器配置

[[email protected] ~]# cat /etc/keepalived/keepalived.conf

! Configuration File for keepalived

 

global_defs {

   notification_email {

     [email protected]

   }

   notification_email_from [email protected]

   smtp_server 127.0.0.1

   smtp_connect_timeout 30

   router_id LVS_DEVEL

}

 

vrrp_script chk_nginx {

    script "/script/sc.sh"

    interval 2

    weight -10

    fall 2  

    rise 2

}

vrrp_instance VI_1 {

    state BACKUP

    interface eth1

    virtual_router_id 51

    mcast_src_ip 192.168.100.131

    priority 99

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.5.200/24

    }

    track_script {

       chk_nginx

    }

}

 

4. 测试down掉master那台服务器的nginx后vip被backup抢占后服务继续不中断

重启keepalived服务

[[email protected] ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:0d:f3:5d brd ff:ff:ff:ff:ff:ff

    inet 192.168.5.129/24 brd 192.168.5.255 scope global eth0

    inet 192.168.5.200/32 scope global eth0（vip被centos-1抢占着）

    inet6 fe80::20c:29ff:fe0d:f35d/64 scope link

       valid_lft forever preferred_lft forever

[[email protected] ~]# curl 192.168.5.200（访问vip地址测试）

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!--------2</h1>

 

</BODY>

</HTML>

 

[[email protected] ~]# curl 192.168.5.200

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!</h1>

 

</BODY>

</HTML>

 

把centos-1的nginx服务down掉以后：

[[email protected] ~]# /usr/local/nginx/sbin/nginx -s stop

[[email protected] ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:0c:29:0d:f3:5d brd ff:ff:ff:ff:ff:ff

    inet 192.168.5.129/24 brd 192.168.5.255 scope global eth0  

    inet6 fe80::20c:29ff:fe0d:f35d/64 scope link （vip已经退出centos-1）

       valid_lft forever preferred_lft forever

[[email protected] ~]# service keepalived status

keepalived 已停

 

[[email protected] ~]# ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

    inet 127.0.0.1/8 scope host lo

    inet6 ::1/128 scope host

       valid_lft forever preferred_lft forever

2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:50:56:2b:42:c1 brd ff:ff:ff:ff:ff:ff

    inet 192.168.5.131/24 brd 192.168.5.255 scope global eth1

    inet 192.168.5.200/32 scope global eth1（vip被centos-2抢占）

    inet6 fe80::250:56ff:fe2b:42c1/64 scope link

       valid_lft forever preferred_lft forever

测试服务是否可继续访问：

 

[[email protected] ~]# curl 192.168.5.200

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!</h1>

 

</BODY>

</HTML>

 

[[email protected] ~]#

[[email protected] ~]# curl 192.168.5.200

<HTML>

<HEAD>

<TITLE>JSP测试页面---HelloWorld!</TITLE>

</HEAD>

<BODY>

<h1>132 Hello World!--------2</h1>

 

</BODY>

</HTML>

 

5.优化nginx+keepalived

因为这样做nginx+keepalived 会抢占资源，这样客户体验就不好，

为了keepalived设置不抢占资源：修改增加参数

两边设置backup

nopreempt（增加这项参数）


本文出自 “成为运维的路上” 博客，请务必保留此出处http://legehappy.blog.51cto.com/13251607/1962797

以上是关于Nginx+Keepalived搭建高可用负载平衡WEB 集群的主要内容，如果未能解决你的问题，请参考以下文章

Nginx+Keepalived(双机热备)搭建高可用负载均衡环境(HA)

Centos7.5 配置 Nginx+Keepalived 搭建高可用负载均衡

公司nginx keepalived tomcat cxf 搭建高可用负载均衡实战系列1- keepalived安装配置

公司nginx keepalived tomcat cxf 搭建高可用负载均衡实战系列1- keepalived的安装配置

LVS+keepalived+nginx搭建高可用负载均衡集群(详细配置步骤讲解)

Linux运维-集群技术进阶Nginx+Keepalived+Tomcat搭建高可用/负载均衡/动静分离的Webserver集群