关于开发中负载均衡(代理)的一些笔记

Posted 2021-12-18 山河已无恙

写在前面

---

• 负载这一块的知识有些乱, 听人家常说软负载、硬负载，四层负载，七层负载、客户端负载,服务端负载之类的,所以梳理一下。
• 负载均衡在系统架构中是一个非常重要,通过负载均衡可以提高系统的高可用，缓解网络、硬件资源的限制。
• 博文主要涉及一些项目中常用的负载方式，很浅，不涉及负载算法啥的，做负载离不来代理，所以文中提到代理服务器即负载服务器。
• 这里梳理的方式从 Dev和Ops的概念出发。这里简单分类一下：
  • 软负载
    • 客户端负载
      • Spring Cloud Ribbon
    • 服务端负载
      • Ngixn (4-7)层负载
      • Hyproxy (4-7)层负载
      • LVS(4)层负载
      • kube-proxy (4-7)层负载
  • 硬负载
    • F5

关于 LVS和kube-proxy、F5我们这里之后在和小伙伴分享，F5没有接触过，LVS的demo容器的方式一直没有成功，kube-proxy这一块我还没学到，只是简单的了解.

如果能深刻理解苦难，苦难就会给人带来崇高感 。 ——路遥

---

一、软负载

处理传输层到应用层的数据，为了能通一个URL将前端的访问分发到后台的多个服务器上

1、客户端负载

Dev 即开发角度的负载均衡。开发中的负载均衡一般是在微服务中涉及。服务提供方一般以多实例的形式提供服务，负载均衡功能能够让服务调用方连接到合适的服务节点。 并且，服务节点选择的过程对服务调用方来说是透明的。

所以这里理解为是客户端的负载均衡，是相对服务端负载均衡而言。

客户端负载均衡来讲,就是调用的客户端本身是知道所有服务信息,当需要调用服务上的接口的时候,客户端从自身所维护的服务列表中,根据提前配置好的负载均衡策略,自己挑选一个服务来调用,此时,客户端知道它所调用的是哪一个服务.

在Spring Cloud 中使用在RestTemplate进行服务调用,要想使用负载均衡功能,需要使用Spring Cloud Ribbon。

Spring Cloud Ribbon是一个基于HTTP和TCP的客户端负载均衡工具,它基于Nettlix Ribbon实现。通过Spring Cloud的封装,可以让我们轻松地将面向服务的REST模板请求自动转换成客户端负载均衡的服务调用。

使用时需要给RestTemplate实例上添加一个@LoadBalanced注解即可,此时, RestTemplate就会自动具备负载均衡功能,这个负载均衡就是客户端负载均衡。

package com.liruilong.consulcon;
 
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;
 
@SpringBootApplication
@EnableDiscoveryClient
public class ConsulConApplication 
 
    public static void main(String[] args) 
        SpringApplication.run(ConsulConApplication.class, args);
    
 
    @Bean
    @LoadBalanced
    RestTemplate restTemplate()
        return  new RestTemplate();
    
 

二、服务端负载

Ops 即运维角度的负载均衡,这里的负载我们也称为服务端负载

所谓服务端负载均衡,比如传统的nginx的方式，调用的客户端并不知道具体是哪个服务提供的服务,它也不关心,反正请求发送给Nginx, 或者hyproxy作为代理的服务器，然后 Ngixn 在请求负载任意服务,客户端只需要记着Nginx的地址即可。

1、Nginx负载

七层(应用层)负载

Nginx 7层负载是最常见的一种负载,所谓7层负载，即应用层负载，即基于应用层协议(TELNET,SSH,HTTP,SMTP,POP…)做的代理，7层负载需要解析数据包的具体内容，需要消耗额外的cpu，然后根据具体内容(url, 参数, cookie, 请求头)匹配相应的路径，然后转发到相应的服务器。转发的过程是：建立和目标机器的连接，然后转发请求，收到响应数据在转发给请求客户端。

使用docker构建一个内部网络

┌──[[e-mail]]-[~]
└─$ docker network create --subnet 10.1.1.1/24 load_balancing
0e0cdf9c70b038f9bcd44fd282ddc3e5bff77403ca28ce5b9006c20793ae2f8d

内网里运行两个httpd服务
10.1.1.22

┌──[[e-mail]]-[~/load_balancing]
└─$ docker run -itd  --name=web1 --net=load_balancing -p 80  -h web1 --ip 10.1.1.22 httpd
ccaa091f295d40c61e50f103e9d84b86caddf9f98d6e5075de3690d93ab48f70
┌──[[e-mail]]-[~/load_balancing]
└─$ echo 10.1.1.22 >index.html;cat index.html
10.1.1.22
┌──[[e-mail]]-[~/load_balancing]
└─$ docker cp ./index.html web1:/usr/local/apache2/htdocs/
┌──[[e-mail]]-[~/load_balancing]
└─$ docker ps
CONTAINER ID   IMAGE     COMMAND              CREATED          STATUS          PORTS                                     NAMES
ccaa091f295d   httpd     "httpd-foreground"   29 seconds ago   Up 28 seconds   0.0.0.0:49153->80/tcp, :::49153->80/tcp   web1
┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:49153
10.1.1.22
┌──[[e-mail]]-[~/load_balancing]
└─$

10.1.1.33

┌──[[e-mail]]-[~/load_balancing]
└─$ docker run -itd  --name=web2 --net=load_balancing -p 80  -h web2 --ip 10.1.1.33 httpd
5b08d54cf4983f6f6ce69cc0cee4b2eab2684cfde8deee89796196760924e434
┌──[[e-mail]]-[~/load_balancing]
└─$  echo 10.1.1.33 >index.html;cat index.html
10.1.1.33
┌──[[e-mail]]-[~/load_balancing]
└─$ docker cp ./index.html web2:/usr/local/apache2/htdocs/
┌──[[e-mail]]-[~/load_balancing]
└─$ docker ps
CONTAINER ID   IMAGE     COMMAND              CREATED          STATUS          PORTS                                     NAMES
5b08d54cf498   httpd     "httpd-foreground"   40 seconds ago   Up 38 seconds   0.0.0.0:49154->80/tcp, :::49154->80/tcp   web2
ccaa091f295d   httpd     "httpd-foreground"   6 minutes ago    Up 6 minutes    0.0.0.0:49153->80/tcp, :::49153->80/tcp   web1
┌──[[e-mail]]-[~/load_balancing]
└─$  curl 127.0.0.1:49154
10.1.1.33
┌──[[e-mail]]-[~/load_balancing]
└─$

Ngixn实现到上面两个httpd服务的负载

ng配置文件

┌──[[e-mail]]-[~/load_balancing]
└─$ cat nginx.conf
user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
#daemon off;
events 
    worker_connections  1024;

http 
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$upstream_addr - $remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" ';
    access_log  /var/log/nginx/nginx_access.log  main;
    sendfile        on;
    keepalive_timeout  65;
    include /etc/nginx/conf.d/*.conf;
    server 
        listen          8099;
        server_name     localhost;
        root            /var/www/html/;
        index           index.html index.htm;
        access_log      /var/log/nginx/default_access.log main;
        error_log       /var/log/nginx/default_error.log;
        location / 
                proxy_pass http://backend;
        
        location ~ .* 
                        proxy_pass http://backend;
                        proxy_set_header Host $http_host;
                        proxy_set_header X-Real-IP $remote_addr;
                        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        

    upstream backend 
        server web2:80;
        server web1:80;
    

运行Nginx容器

┌──[[e-mail]]-[~/load_balancing]
└─$ docker run -itd -p 8099:8099 --name=nginx --network=load_balancing  -v $PWD/nginx.conf:/etc/nginx/nginx.conf nginx
0af20ed5c390e81398037a498fc7d385cac96cd2f403a8b08f6f4e09d7a20ee0

测试一下

┌──[[e-mail]]-[~/load_balancing]
└─$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                               NAMES
0af20ed5c390   nginx     "/docker-entrypoint.…"   4 seconds ago    Up 3 seconds    80/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp   nginx
b16bcb89e0a8   httpd     "httpd-foreground"       32 minutes ago   Up 32 minutes   0.0.0.0:49155->80/tcp, :::49155->80/tcp             web1
5b08d54cf498   httpd     "httpd-foreground"       39 minutes ago   Up 39 minutes   0.0.0.0:49154->80/tcp, :::49154->80/tcp             web2
┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:8099
10.1.1.33
┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:8099
10.1.1.22
┌──[[e-mail]]-[~/load_balancing]
└─$

四层(传输层)负载

所谓四层负载，即在传输层协议的基础上来做负载，基于TCP,UDP等协议，传输层的作用是确保数据被可靠的传输送到目标地址，能够让应用程序之间实现通信，所以彼此传递的是数据包，标识的只有IP+端口。不涉及具体的url其他结构解析。路径匹配等，不会涉及具体的应用层协议，所以理论上四层负载要比七成负载快。

nginx 四层代理是nginx1.9.0开始新增的功能，需要开启--with-stream模块，可以实现四层协议的转发、代理、负载等功能。

这里的话，我们还是用容器的方式。配置方式和七层主要是配置文件的区别

ng配置文件

└─$ cat nginx.conf
user  nginx;
worker_processes  auto;
error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;
daemon off;

events 
    worker_connections  1024;

# 四层代理的方式
stream
    server 
        listen     8088;
        proxy_pass backend;
        
    upstream backend 
        server web1:80;
        server web2:80;
    




http  # 这个是协议级别
    include mime.types;
    default_type application/octet-stream;
    keepalive_timeout 65;
    gzip on;
    server  #这个是服务器级别
        listen 80;
        server_name localhost;
        location /  #这个是请求级别
            root html;
            index index.html index.htm;
        
    

启动4层负载的Nginx

┌──[[e-mail]]-[~/load_balancing]
└─$ docker run -itd -p 8088:8088 --name=nginx4 --network=load_balancing  -v $PWD/nginx.conf:/etc/nginx/nginx.conf nginx nginx
7c342f86752c5fe494b5a142983503d82dd11ea54e2968da7172f2201d1c45ea
┌──[[e-mail]]-[~/load_balancing]
└─$ docker logs nginx4
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/11/16 16:25:50 [notice] 1#1: using the "epoll" event method
2021/11/16 16:25:50 [notice] 1#1: nginx/1.21.3
2021/11/16 16:25:50 [notice] 1#1: built by gcc 8.3.0 (Debian 8.3.0-6)
2021/11/16 16:25:50 [notice] 1#1: OS: Linux 3.10.0-693.el7.x86_64
2021/11/16 16:25:50 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/11/16 16:25:50 [notice] 1#1: start worker processes
2021/11/16 16:25:50 [notice] 1#1: start worker process 31
2021/11/16 16:25:50 [notice] 1#1: start worker process 32
┌──[[e-mail]]-[~/load_balancing]
└─$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                               NAMES
7c342f86752c   nginx     "/docker-entrypoint.…"   21 seconds ago   Up 19 seconds   80/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp   nginx4
0af20ed5c390   nginx     "/docker-entrypoint.…"   2 hours ago      Up 2 hours      80/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp   nginx
b16bcb89e0a8   httpd     "httpd-foreground"       2 hours ago      Up 2 hours      0.0.0.0:49155->80/tcp, :::49155->80/tcp             web1
5b08d54cf498   httpd     "httpd-foreground"       2 hours ago      Up 2 hours      0.0.0.0:49154->80/tcp, :::49154->80/tcp             web2

测试一下

┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:8088
10.1.1.22
┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:8088
10.1.1.33
┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:8088
10.1.1.22

2、HAProxy负载

HAProxy 是一款提供高可用性、负载均衡以及基于TCP（第四层）和HTTP（第七层）应用的代理软件，支持虚拟主机，它是免费、快速并且可靠的一种解决方案。

HAProxy特别适用于那些负载特大的web站点，这些站点通常又需要会话保持或七层处理。HAProxy完全可以支持数以万计的并发连接。

7 层(应用层)负载

这里我们还用之前的连个httpd服务演示

┌──[[e-mail]]-[~/load_balancing]
└─$ docker pull haproxy

haproxy.cfg配置文件

┌──[[e-mail]]-[~/load_balancing]
└─$ cat haproxy.cfg
global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 4096

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000ms
    timeout client  50000ms
    timeout server  50000ms

listen  admin_stats
    bind 0.0.0.0:8070
    mode http
    stats enable
    stats hide-version
    stats scope .
    stats realm Haproxy\\ Statistics
    stats refresh 30s
    stats uri /
    stats auth user:pass

frontend balancer
    bind 0.0.0.0:8077
    mode http
    default_backend web_backends

backend web_backends
    mode http
    option forwardfor
    balance roundrobin
    server web1 web1:80 check
    server web2 web2:80 check
    option httpchk GET /
    http-check expect status 200


┌──[[e-mail]]-[~/load_balancing]
└─$

┌──[[e-mail]]-[~/load_balancing]
└─$ docker run -itd -p 8070 -p 8077 --name=haproxy --net=load_balancing -v $PWD/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg  haproxy
a467636f6fdc75a7cf7538c32caab78c5a9e465235e2baec212bc9274a2cd534
┌──[[e-mail]]-[~/load_balancing]
└─$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS          PORTS                                                                                      NAMES
a467636f6fdc   haproxy   "docker-entrypoint.s…"   2 seconds ago    Up 1 second     0.0.0.0:49163->8070/tcp, :::49163->8070/tcp, 0.0.0.0:49162->8077/tcp, :::49162->8077/tcp   haproxy
7c342f86752c   nginx     "/docker-entrypoint.…"   40 minutes ago   Up 40 minutes   80/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp                                          nginx4
0af20ed5c390   nginx     "/docker-entrypoint.…"   2 hours ago      Up 2 hours      80/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp                                          nginx
b16bcb89e0a8   httpd     "httpd-foreground"       3 hours ago      Up 3 hours      0.0.0.0:49155->80/tcp, :::49155->80/tcp                                                    web1
5b08d54cf498   httpd     "httpd-foreground"       3 hours ago      Up 3 hours      0.0.0.0:49154->80/tcp, :::49154->80/tcp 

测试下

┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:49162
10.1.1.22
┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:49162
10.1.1.33
┌──[[e-mail]]-[~/load_balancing]
└─$                                                  

统计页面

4 层(传输层)负载

四层负载和七层负载也是配置文件的区别

┌──[[e-mail]]-[~/load_balancing]
└─$ cat haproxy.cfg
global
    log 127.0.0.1 local0
    log 127.0.0.1 local1 notice
    maxconn 4096

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000ms
    timeout client  50000ms
    timeout server  50000ms

listen  admin_stats
    bind 0.0.0.0:8070
    mode http
    stats enable
    stats hide-version
    stats scope .
    stats realm Haproxy\\ Statistics
    stats refresh 30s
    stats uri /
    stats auth user:pass

listen web
    bind *:3306
    mode tcp
    balance roundrobin
    server web1 10.1.1.22:80  weight 1  check inter 1s rise 2 fall 2
    server web2 10.1.1.33:80  weight 1  check inter 1s rise 2 fall 2

┌──[[e-mail]]-[~/load_balancing]
└─$

运行容器并测试

┌──[[e-mail]]-[~/load_balancing]
└─$ docker run -itd -p 8070 -p 3306 --name=haproxy4 --net=load_balancing -v $PWD/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg  haproxy
602eebc6ad525c420bad6061d9465fa7cdf2036d7db177efaf5450edf8c9db8b
┌──[[e-mail]]-[~/load_balancing]
└─$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS
                  NAMES
602eebc6ad52   haproxy   "docker-entrypoint.s…"   8 seconds ago   Up 6 seconds   0.0.0.0:49165->3306/tcp, :::49165->3306/tcp, 0.0.0.0:49164->8070/tcp, :::49164->8070/tcp   haproxy4
a467636f6fdc   haproxy   "docker-entrypoint.s…"   19 hours ago    Up 19 hours    0.0.0.0:49163->8070/tcp, :::49163->8070/tcp, 0.0.0.0:49162->8077/tcp, :::49162->8077/tcp   haproxy
7c342f86752c   nginx     "/docker-entrypoint.…"   20 hours ago    Up 20 hours    80/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp
                  nginx4
0af20ed5c390   nginx     "/docker-entrypoint.…"   21 hours ago    Up 21 hours    80/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp
                  nginx
b16bcb89e0a8   httpd     "httpd-foreground"       22 hours ago    Up 22 hours    0.0.0.0:49155->80/tcp, :::49155->80/tcp
                  web1
5b08d54cf498   httpd     "httpd-foreground"       22 hours ago    Up 22 hours    0.0.0.0:49154->80/tcp, :::49154->80/tcp
                  web2
┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:49165
10.1.1.22
┌──[[e-mail]]-[~/load_balancing]
└─$ curl 127.0.0.1:49165
10.1.1.33
┌──[[e-mail]]-[~/load_balancing]
└─$