nginx内置高可用配置与第三方高可用模块nginx_ustream_check_mudule配置

Posted 2021-02-28 陈雷雷

1. nginx 第三方高可用模块

IP	备注
10.0.0.63	proxy
10.0.0.64	web1
10.0.0.65	web2

这里会讲解一些nignx常用高可用方案,以及引入第三方高可用模块来了解nginx作为高可用服务它是如何运行的

2. 基础环境安装

10.0.0.63  nginx 增加 nginx_upstream_check_module模块
10.0.0.64  nginx
10.0.0.65  nginx 
-------------------------------------------------------------
#3台都安装上nginxx:
useradd  www -u 1200 -M -s /sbin/nologin
mkdir -p /var/log/nginx
yum install -y cmake pcre pcre-devel openssl openssl-devel gd-devel \\
    zlib-devel gcc gcc-c++ net-tools iproute telnet wget curl &&\\
    yum clean all && \\
    rm -rf /var/cache/yum/*
mkdir -p /server/tools
cd /server/tools
wget https://www.chenleilei.net/soft/nginx-1.16.1.tar.gz
tar xf nginx-1.16.1.tar.gz
cd nginx-1.16.1
./configure --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www \\
    --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module \\
    --pid-path=/var/run/nginx/nginx.pid
make -j 4 && make install && \\
    rm -rf /usr/local/nginx/html/*  && \\
    echo "nginx daemo hello" >/usr/local/nginx/html/index.html  && \\
echo "export PATH=$PATH:/usr/local/nginx/sbin" >>/etc/profile
chown -R www.www /var/log/nginx /usr/local/nginx
source /etc/profile
nginx -s reload
ps -ef|grep nginx

3. nginx的高可用配置

默认的nginx高可用是一个后端连接不可用时且没有健康检查,他会一台台轮询.在 如: 金钱交易的期间都会关闭这个默认高可用\'proxy_next_upstream\',原因就是,敏感交易有可能在默认时间未结束前得到了响应,而另一方面有其他服务器响应了该请求,可能会出现重复交易情况,所以建议关闭默认的高可用策略,从而改用更简单易用的其他高可用模块来实现,它就是 "ngx_http_proxy_module"

3.1 默认高可用策略介绍

server 10.0.0.10:80 max_fails=2 fail_timeout=30;
weight        权重
max_fails     失败次数后停止该服务器
fail_timeout  踢出后重新检测时间. 一般max_fails和fail_timeout一起启用的,也就是失败n次,等待N秒
backup        备用服务器
max_conns     允许最大连接数
slow_start    节点恢复后不立即加入到集群.

3.2 安装 ngx_upstream_check_module [10.0.0.63]

github: https://github.com/yaoweibin/nginx_upstream_check_module

[root@master tools]# git clone https://github.com/yaoweibin/nginx_upstream_check_module.git
[root@master tools]# ll
drwxr-xr-x 7 root root      4096 Apr 16 21:53 nginx_upstream_check_module
[root@master tools]# cd nginx_upstream_check_module/
[root@master nginx_upstream_check_module]# pwd
/server/tools/nginx_upstream_check_module   #这个路径用于编译nginx新增模块使用.


# nginx_upstream_check_module 模块目录:
[root@master nginx_upstream_check_module]# ll
total 268
-rw-r--r-- 1 root root      0 Apr 16 21:53 CHANGES
-rw-r--r-- 1 root root   7921 Apr 16 21:53 check_1.11.1+.patch
-rw-r--r-- 1 root root   8330 Apr 16 21:53 check_1.11.5+.patch
-rw-r--r-- 1 root root   8060 Apr 16 21:53 check_1.12.1+.patch
-rw-r--r-- 1 root root   8054 Apr 16 21:53 check_1.14.0+.patch
-rw-r--r-- 1 root root   8409 Apr 16 21:53 check_1.16.1+.patch
-rw-r--r-- 1 root root   5483 Apr 16 21:53 check_1.2.1.patch
-rw-r--r-- 1 root root   7130 Apr 16 21:53 check_1.2.2+.patch
-rw-r--r-- 1 root root   7094 Apr 16 21:53 check_1.2.6+.patch
-rw-r--r-- 1 root root   6791 Apr 16 21:53 check_1.5.12+.patch
-rw-r--r-- 1 root root   8295 Apr 16 21:53 check_1.7.2+.patch
-rw-r--r-- 1 root root   8346 Apr 16 21:53 check_1.7.5+.patch
-rw-r--r-- 1 root root   8509 Apr 16 21:53 check_1.9.2+.patch
-rw-r--r-- 1 root root   6943 Apr 16 21:53 check.patch
-rw-r--r-- 1 root root    749 Apr 16 21:53 config
drwxr-xr-x 2 root root     43 Apr 16 21:53 doc
-rw-r--r-- 1 root root   1709 Apr 16 21:53 nginx-sticky-module.patch
drwxr-xr-x 2 root root     29 Apr 16 21:53 nginx-tests
-rw-r--r-- 1 root root 112085 Apr 16 21:53 ngx_http_upstream_check_module.c
-rw-r--r-- 1 root root    529 Apr 16 21:53 ngx_http_upstream_check_module.h
-rw-r--r-- 1 root root   2848 Apr 16 21:53 ngx_http_upstream_jvm_route_module.patch
-rw-r--r-- 1 root root  11509 Apr 16 21:53 README
drwxr-xr-x 6 root root     79 Apr 16 21:53 test
-rw-r--r-- 1 root root   3342 Apr 16 21:53 upstream_fair.patch
drwxr-xr-x 2 root root     81 Apr 16 21:53 util


#编译nginx:
1. 获取nginx原有编译参数: 
configure arguments: --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid

2. 同步模块代码:
cd /server/tools/nginx-1.16.1
 patch -p1 < /server/tools/nginx_upstream_check_module/check_1.16.1+.patch
输出:
[root@master nginx-1.16.1]# patch -p1 < /server/tools/nginx_upstream_check_module/check_1.16.1+.patch
patching file src/http/modules/ngx_http_upstream_hash_module.c
patching file src/http/modules/ngx_http_upstream_ip_hash_module.c
patching file src/http/modules/ngx_http_upstream_least_conn_module.c
patching file src/http/ngx_http_upstream_round_robin.c
patching file src/http/ngx_http_upstream_round_robin.h

3. 重新编译nginx来添加模块:
[root@master ~ ]# cd /server/tools/nginx-1.16.1
[root@master nginx-1.16.1]# ./configure --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid --add-module=/server/tools/nginx_upstream_check_module/

[root@master nginx-1.16.1]# make 
make结束后拷贝新的nginx二进制文件替换老版本,这和版本升级流程一样

#拷贝二进制文件替换老的nginx二进制文件
[root@master nginx-1.16.1]# \\cp -a objs/nginx /usr/local/nginx/sbin/

#查看进程:
[root@master nginx-1.16.1]# ps -ef|grep nginx
root      43754      1  0 22:17 ?        00:00:00 nginx: master process nginx
www       43755  43754  0 22:17 ?        00:00:00 nginx: worker process
root      43774  37915  0 22:17 pts/0    00:00:00 grep --color=auto nginx

#关闭nginx后重新启动nginx:
[root@master nginx-1.16.1]# pkill nginx
[root@master nginx-1.16.1]# nginx

#检查模块安装是否成功:
[root@master nginx-1.16.1]# nginx -V
nginx version: nginx/1.16.1
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) 
built with OpenSSL 1.0.2k-fips  26 Jan 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/local/nginx --with-http_image_filter_module --user=www --group=www --with-http_ssl_module --with-http_v2_module --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid --add-module=/server/tools/nginx_upstream_check_module/

#可以看到已经安装了

顺便做一下语法高亮显示的小优化:
mkdir ~/.vim -p
cp -r contrib/vim/* ~/.vim/

4.第三方高可用模块nginx_ustream_check_mudule的配置使用

安装完了模块后,需要对nginx进行配置才能使得新模块得到应用

egrep -v "#|^$" /usr/local/nginx/conf/nginx.conf.default > /usr/local/nginx/conf/nginx.conf
vim /usr/local/nginx/conf/nginx.conf

配置为如下:
#-----------------------------------------------------------------------------#
worker_processes  1;
events {
    worker_connections  1024;
}
http {
log_format  main  \'$remote_addr - $remote_user [$time_local] "$request" \'
                  \'$status $body_bytes_sent "$http_referer" \'
                  \'"$http_user_agent" "$http_x_forwarded_for"\';
access_log  logs/access.log  main;
upstream web {
        server 10.0.0.64;
        server 10.0.0.65;
        }
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  locaalhost;
        location / {
                proxy_connect_timeout 2s;
                proxy_read_timeout 2s;
                proxy_next_upstream off;
                proxy_pass http://web;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}
#-----------------------------------------------------------------------------#


2. 修改添加模块配置项:
worker_processes  1;
events {
    worker_connections  1024;
}
http {
log_format  main  \'$remote_addr - $remote_user [$time_local] "$request" \'
                  \'$status $body_bytes_sent "$http_referer" \'
                  \'"$http_user_agent" "$http_x_forwarded_for"\';
access_log  logs/access.log  main;
upstream web {
        server 10.0.0.64;
        server 10.0.0.65;
        #模块配置项 用于检查后端服务器是否正常
        check interval=5000 rise=1 fall=3 timeout=4000;
        check interval=3000 rise=2 fall=5 timeout=1000 type=ssl_hello;
        check interval=3000 rise=2 fall=5 timeout=1000 type=http;
        check_http_send "HEAD / HTTP/1.0\\r\\n\\r\\n";
        check_http_expect_alive http_2xx http_3xx http_4xx http_5xx;
        #模块配置项 用于检查后端服务器是否正常
        }
    include       mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server {
        listen       80;
        server_name  locaalhost;
        location / {
                proxy_connect_timeout 2s;
                proxy_read_timeout 2s;
                proxy_next_upstream off;
                proxy_pass http://web;
        }
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   html;
        }
    }
}

验证模块可用性:

1. 为后端nginx添加时别内容:

10.0.0.64服务器添加:
echo  "10.0.0.64 nginx daemon" >index.html 

10.0.0.65服务器添加:
echo  "10.0.0.65 nginx daemon" >index.html 

可以看出节点分配正常

4.1 后端健康检查测试

前端负载均衡 开启日志记录,获取访问的后端节点IP,便于查看配置是否正确,这个配置在企业中也是必备,务必掌握

http模块下修改默认日志记录为如下:

log_format  main  \'$remote_addr - $remote_user [$time_local] "$request" \'
                  \'$status $body_bytes_sent "$http_referer" \'
                  \'"$http_user_agent" "$http_x_forwarded_for" $request_time $upstream_response_time $upstream_addr $upstream_status\';

access_log  logs/access.log  main;

配置完毕后,继续测试负载均衡模块的可用性

默认端口都开 检查日志访问情况:
10.0.0.1 - - [16/Apr/2020:23:11:29 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.002 10.0.0.64:80 200
10.0.0.1 - - [16/Apr/2020:23:11:29 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.001 10.0.0.65:80 200
10.0.0.1 - - [16/Apr/2020:23:11:30 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.002 10.0.0.64:80 200
10.0.0.1 - - [16/Apr/2020:23:11:32 +0800] "GET / HTTP/1.1" 200 23 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36" "-" 0.001 0.001 10.0.0.65:80 200

#从这里可以看出,我们访问10.0.0.63负载均衡的时候,他会一次去访问10.0.0.64,一次去访问10.0.0.65, 照此依次轮询.
#现在关闭后端某个节点,用于测试模块是否正常检测:

2020/04/16 23:18:38 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:41 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:44 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:47 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:50 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:18:50 [error] 46559#0: disable check peer: 10.0.0.64:80   #<---- 移除了该节点

他会进行5次检测,发现后端节点无法访问,直接将故障节点移除集群.
同时当这个集群角色恢复后,该模块依然能够检测出并将其重新加入集群.
如下:
2020/04/16 23:20:53 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:20:56 [error] 46559#0: send() failed (111: Connection refused)
2020/04/16 23:21:03 [error] 46559#0: enable check peer: 10.0.0.64:80    #<---- 添加了该节点