consul+upsync+nginx实现动态负载均衡

Posted 2020-12-10

consul+upsync+nginx实现动态负载均衡

原理：

将upstream配置与Nginx本身解耦，实现在线修改upstream信息nginx动态生效。

优势：1、无需登录服务器配置

        2、避免nginx进行reload

        3、在线配置，后期更容易实现蓝绿。与Apollo这种类似，但需要借助upsync模块

安装nginx支持upsync：

测试: nginx version: nginx/1.12.2 可行

--add-module=/tmp/nginx-upsync-module

cd /tmp/
git clone https://github.com/weibocom/nginx-upsync-module.git

./configure --prefix=/data1/nginx/release/{{dir_date}} --conf-path=/data1/nginx/release/{{dir_date}}/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --user=nginx --group=nginx --with-http_ssl_module --with-http_v2_module --with-http_dav_module --with-http_stub_status_module --with-threads --with-file-aio --with-stream --with-stream_ssl_module --add-module=/tmp/nginx-upsync-module

安装配置consul

mkdir /data1/consul
cd /data1/consul
wget https://releases.hashicorp.com/consul/1.7.3/consul_1.7.3_linux_amd64.zip
unzip consul_1.7.3_linux_amd64.zip

## 启动consul  ##应为集群
consul agent --server --bind=172.16.2.128 --data-dir=/tmp/consule -bootstrap-expect=1 -node=nginx-lb -client 0.0.0.0 -ui

新增一条upstream信息

## 新增一个upstream信息
curl -X PUT http://172.16.2.128:8500/v1/kv/upstreams/test/172.16.2.129:8086

配置nginx配置文件

upstream test_server {  
    server 127.0.0.1:11111;
    upsync 172.16.2.128:8500/v1/kv/upstreams/test upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
    upsync_dump_path /data1/nginx/current/consul/test.ppdapi.com.conf;
}

server {
    listen 80;
    server_name   test.ppdai.com;

    index default.htm index.html index.htm default.html;
    gzip on;

    location / {
        proxy_pass       http://test_server;
    }

}

consul的日常管理

关于upsync模块学习：https://github.com/weibocom/nginx-upsync-module#upsync_timeout

触发新增：
curl -X PUT http://172.16.2.128:8500/v1/kv/upstreams/test/172.16.2.129:8086
触发删除：
curl -X DELETE http://172.16.2.128:8500/v1/kv/upstreams/test/172.16.2.129:8086
修改权重：
http://172.16.2.128:8500/v1/kv/upstreams/test/172.16.2.129:8081
json：{"weight":1, "max_fails":2, "fail_timeout":10}
curl -X PUT -d ‘{"weight":1, "max_fails":2, "fail_timeout":10}‘ http://$consul_ip:$port/v1/kv/$dir1/$upstream_name/$backend_ip:$backend_port

拍拍贷consul配置信息参考

##tsource.ppdapi.com###

upstream tsource_server {  
    server 127.0.0.1:11111;
    upsync 10.0.24.251:8500/v1/kv/upstreams/tsource_server upsync_timeout=6m upsync_interval=500ms upsync_type=consul strong_dependency=off;
    upsync_dump_path /data/nginx/conf/tsource.ppdapi.com.conf;

    check interval=3000 rise=2 fall=5 timeout=1000 type=http;
    check_http_send "HEAD http://tsource.ppdapi.com/hs HTTP/1.0

";
    check_http_expect_alive http_2xx http_3xx ;

}

server {
    listen 80;
    server_name    tsource.ppdapi.com;

    server_tokens   off;
    proxy_hide_header X-Powered-By;
    proxy_hide_header X-AspNet-Version;
    index default.htm index.html index.htm default.html;
    gzip on;

    location / {
        proxy_pass       http://tsource_server;
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
    }

}