haproxy动态增减主机与keepalived高级应用

Posted 2020-11-15 maxuebin

一：本文将详细介绍haproxy的配置使用以及高级功能的使用，比如通过haproxy进行动态添加删除负载集群中的后端web服务器的指定主机，另外将详细介绍keepalived的详细配置方法、配置实例及实现方式，具体如下：

1.1：通过haproxy动态增减web集群中的主机，打开haproxy的主机在线维护功能，主要原理是haproxy监听本地socket文件，mysql写localhost会查抄默认路径为/var/lib/mysql/mysql.socket文件，但是使用IP地址是使用的TCP协议访问，就不需要查找socket文件了，所以使用localhost连接本机MySQL和写127.0.0.1连接本机MySQL的实现方式是不一样的，haproxy配置socket文件路径：

1.1.1：haproxy配置文件添加socket配置信息如下：

[root@localhost ~]# cat /etc/haproxy/haproxy.cfg 
global
maxconn 100000
chroot /usr/local/haproxy
uid 99
gid 99
daemon
nbproc 1
pidfile /usr/local/haproxy/logs/haproxy.pid
log 127.0.0.1 local0 info
stats socket /var/lib/haproxy/haproxy.sock mode 600 level admin #设置socket文件路径，权限为600级别为admin，还有其他级别，但是动态添加主机需要admin权限

1.1.2：创建保存socket文件的目录：

# mkdir  /var/lib/haproxy

1.1.3：重启haproxy服务：

[root@localhost ~]# /etc/init.d/haproxy  restart
Restarting haproxy (via systemctl):                        [  OK  ]

1.1.4：验证haproxy的socket文件：

[root@localhost ~]# ll /var/lib/haproxy/
total 0
srw------- 1 root root 0 Dec 31 21:02 haproxy.socket

1.2：安装socat命令，socat是和socket通信的命令：

1.2.1：安装命令：

[root@localhost ~]# yum install socat

1.2.2：命令使用帮助：

[root@localhost ~]# echo "help" | socat stdio /var/lib/haproxy/haproxy.socket 
Unknown command. Please enter one of the following commands only :
  help           : this message
  prompt         : toggle interactive mode with prompt
  quit           : disconnect
  disable agent  : disable agent checks (use ‘set server‘ instead)
  disable health : disable health checks (use ‘set server‘ instead)
  disable server : disable a server for maintenance (use ‘set server‘ instead)
  enable agent   : enable agent checks (use ‘set server‘ instead)
  enable health  : enable health checks (use ‘set server‘ instead)
  enable server  : enable a disabled server (use ‘set server‘ instead)
  set maxconn server : change a server‘s maxconn setting
  set server     : change a server‘s state, weight or address
  get weight     : report a server‘s current weight
  set weight     : change a server‘s weight (deprecated)
  disable frontend : temporarily disable specific frontend
  enable frontend : re-enable specific frontend
  set maxconn frontend : change a frontend‘s maxconn setting
  show servers state [id]: dump volatile server information (for backend <id>)
  show backend   : list backends in the current running config
  shutdown frontend : stop a specific frontend
  clear table    : remove an entry from a table
  set table [id] : update or create a table entry‘s data
  show table [id]: report table usage stats or dump this table‘s contents
  show errors    : report last request and response errors for each proxy
  clear counters : clear max statistics counters (add ‘all‘ for all counters)
  show info      : report information about the running process
  show stat      : report counters for each proxy and server
  show sess [id] : report the list of current sessions or dump this session
  shutdown session : kill a specific session
  shutdown sessions server : kill sessions on a server
  show pools     : report information about the memory pools usage
  add acl        : add acl entry
  clear acl <id> : clear the content of this acl
  del acl        : delete acl entry
  get acl        : report the patterns matching a sample for an ACL
  show acl [id]  : report available acls or dump an acl‘s contents
  add map        : add map entry
  clear map <id> : clear the content of this map
  del map        : delete map entry
  get map        : report the keys and values matching a sample for a map
  set map        : modify map entry
  show map [id]  : report available maps or dump a map‘s contents
  show stat resolvers [id]: dumps counters from all resolvers section and
                          associated name servers
  set maxconn global : change the per-process maxconn setting
  set rate-limit : change a rate limiting value
  set timeout    : change a timeout setting
  show env [var] : dump environment variables known to the process
  show tls-keys [id|*]: show tls keys references or dump tls ticket keys when id specifie

1.2.3：查看haproxy的详细信息：

[root@localhost ~]# echo "show info" | socat stdio /var/lib/haproxy/haproxy.socket 
Name: HAProxy
Version: 1.7.1
Release_date: 2016/12/13
Nbproc: 1
Process_num: 1
Pid: 20077
Uptime: 0d 0h04m49s
Uptime_sec: 289
Memmax_MB: 0
PoolAlloc_MB: 0
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 200034
Maxsock: 200034
Maxconn: 100000
Hard_maxconn: 100000
CurrConns: 0
CumConns: 3
CumReq: 3
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 0
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 0
SessRate: 0
SessRateLimit: 0
MaxSessRate: 0
SslRate: 0
SslRateLimit: 0
MaxSslRate: 0
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 0
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 7
Run_queue: 1
Idle_pct: 100
node: localhost

1.3：关闭一个后端服务器中的web服务器并在管理界面进行验证：

1.3.1：动态关闭一台web服务器：

[root@localhost ~]# echo "disable server web_port/web1" | socat stdio /var/lib/haproxy/haproxy.socket  #关闭指定backend下的某台web服务器

1.3.2：在haproxy的管理端进行验证该web服务器是否从负载中下线：

1.3.3：动态开启服务器：

[root@localhost ~]# echo "enable  server web_port/web1" | socat stdio /var/lib/haproxy/haproxy.socket

1.3.4：验证服务器是否开启：

二：Keepalived 应用：

官方下载地址：http://www.keepalived.org/download.html

Keepalived权威指南：Keepalived权威指南
2.1：keepalived是VRRP协议在linux系统上的完美实现的解决方案，详细如下：

    虚拟路由冗余协议(Virtual Router Redundancy Protocol，简称VRRP)是由IETF提出的解决局域网中配置静态网关出现单点失效现象的路由协议，1998年已推出正式的RFC2338协议标准。VRRP广泛应用在边缘网络中，它的设计目标是支持特定情况下IP数据流量失败转移不会引起混乱，允许主机使用单路由器，以及及时在实际第一跳路由器使用失败的情形下仍能够维护路由器间的连通性。VRRP是一种选择协议，它可以把一个虚拟路由器的责任动态分配到局域网上的 VRRP 路由器中的一台。控制虚拟路由器 IP 地址的 VRRP 路由器称为主路由器，它负责转发数据包到这些虚拟 IP 地址。[1]  一旦主路由器不可用，这种选择过程就提供了动态的故障转移机制，这就允许虚拟路由器的 IP 地址可以作为终端主机的默认第一跳路由器。是一种LAN接入设备备份协议。一个局域网络内的所有主机都设置缺省网关，这样主机发出的目的地址不在本网段的报文将被通过缺省网关发往三层交换机，从而实现了主机和外部网络的通信。
VRRP是一种路由容错协议，也可以叫做备份路由协议。一个局域网络内的所有主机都设置缺省路由，当网内主机发出的目的地址不在本网段时，报文将被通过缺省路由发往外部路由器，从而实现了主机与外部网络的通信。当缺省路由器down掉（即端口关闭）之后，内部主机将无法与外部通信，如果路由器设置了VRRP时，那么这时，虚拟路由将启用备份路由器，从而实现全网通信。
脑裂是backup收不到master的心跳组播包，可以使用网线直连做心跳。

2.2：部署两台服务器，各安装haproxy和keepalived，实现负载高可用：

2.2.1：haproxy安装方式在上一篇博客有详细介绍：

2.2.2：编译安装keepalived：

 

# tar xvf keepalived-1.2.24.tar.gz
# cd keepalived-1.2.24
# ./configure  --prefix=/usr/local/keepalived-1.2.24
# make && make install
# ln -sv /usr/local/keepalived-1.2.24/ /usr/local/keepalived
# cp keepalived/etc/init.d/keepalived.sysconfig  /etc/sysconfig/keepalived
# cp keepalived/etc/init.d/keepalived.init  /etc/init.d/keepalive
# chmod  a+x /etc/init.d/keepalived
# mkdir  /etc/keepalived 
# ln -sv /usr/local/keepalived/sbin/keepalived   /usr/bin/ #不做软连接报错，提示/bin/bash/keepakived找不到

2.2.3：server1 编辑keepalived配置文件：

! Configuration File for keepalived

global_defs {
  notification_email {  #指定keepalived在发生事件，比如切换IP的时候需要发生邮件的地址，可以有多个，一行一个即可 
    [e-mail]
    [e-mail]
    [e-mail]
  }
  notification_email_from [e-mail]  #指定smtp服务器设置 
  smtp_server 192.168.200.1
  smtp_connect_timeout 30
  router_id LVS_DEVEL  #运行keepalived的标示，可以写主机名等信息 
}

vrrp_instance IP1 {   #监听一组VIP 
    state BACKUP #指定初始状态为backup即备份状态
    interface br0 #监听的本地网卡名称,此处是一块桥接网卡，
    virtual_router_id 10  #路由id，多个网卡环境下，id一定不能一样，范围为10-255 
    priority 90 #优先级，高的为master 
    advert_int 1  #进行通告 
    authentication { #配置认证机制 
        auth_type PASS #认证类型为密码 
        auth_pass 5555 #认证密码
    }
    virtual_ipaddress {
        192.168.10.100 
    }
}
vrrp_instance IP2 {
    state MASTER #第二组IP初始状态是master
    interface br0
    virtual_router_id 20
    priority 100
    advert_int 1
    authentication {
         auth_type PASS
         auth_pass 1111
    }
    virtual_ipaddress {
         192.168.10.101 #具体的虚拟IP地址，两个keepalived设置要一样，否则服务不生效
     }
}

2.3.4：server 2的keepalived配置：

# cat /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
  notification_email {
    [e-mail]
    [e-mail]
    [e-mail]
  }
  notification_email_from [e-mail]
  smtp_server 192.168.200.1
  smtp_connect_timeout 30
  router_id LVS_DEVEL
}

vrrp_instance IP1 {
    state MASTER  #服务器1的master
    interface eth1
    virtual_router_id 10
    priority 150
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 5555
    }
    virtual_ipaddress {
        192.168.10.100 
    }
}
vrrp_instance IP2 {
    state BACKUP #是服务器1的备份
    interface eth1
    virtual_router_id 20
    priority 50
    advert_int 1
    authentication {
         auth_type PASS
         auth_pass 1111
    }
    virtual_ipaddress {
         192.168.10.101
     }
}

2.4：关闭一个keepalived服务，在日志记录中验证IP地址是否会迁移到backup服务器：

2.4.1：在server 1关闭keepalived：

[root@localhost keepalived-1.2.24]# /etc/init.d/keepalived  stop
Stopping keepalived (via systemctl):                       [  OK  

2.4.2：在server 2查看日志：

2.4.3：验证server 2的IP地址：

2.4.4：开启server 1的keepalived：

[root@localhost keepalived-1.2.24]# /etc/init.d/keepalived  start
Starting keepalived (via systemctl):                       [  OK  ]

2.4.5：验证server 2的日志：

2.4.6：验证server 2的IP地址：

2.4.7：验证server 1的ip地址：

2.4.8：使用tcpdump抓取心跳检测的数据包：

2.5：关于keepalived的不抢占：

抢占发生在Master和Backup环境中Master 出现问题导致虚拟IP漂移到了Backup，但是当Master恢复之后IP地址会再次漂移到Master上，并且反反复复出现这样的问题，从而导致出现出现频繁网络切换，而可能引起未知的网络问题，因此可以配置不抢占，当IP漂移到了Backup之后不再往回迁IP，需要手动将IP配置回去才可以，具体如下：

2.5.1：Server1的不抢占的详细配置：

[root@localhost keepalived-1.2.24]# cat  /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
  notification_email {
    [e-mail]
    [e-mail]
    [e-mail]
  }
  notification_email_from [e-mail]
  smtp_server 192.168.200.1
  smtp_connect_timeout 30
  router_id LVS_DEVEL
}

vrrp_instance IP1 {
    state BACKUP
    interface br0
    virtual_router_id 10
    priority 90
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 5555
    }
    virtual_ipaddress {
        192.168.10.100 
    }
}
vrrp_instance IP2 {
    state BACKUP #不抢占需要配置在BACKUP组内，并且优先级高于备份服务器的keepalived组内
    interface br0
    virtual_router_id 20
    nopreempt #声明不抢占，即当IP飘走之后不再根据优先级抢占IP，但是当对方服务出现问题依然可以绑定IP到本地
    priority 100 #配置在高优先级的keepalived组内，低优先级的就不需要配置抢占了
    advert_int 1
    authentication {
         auth_type PASS
         auth_pass 1111
    }
    virtual_ipaddress {
         192.168.10.101
     }
}

2.5.2：Serevr 2不抢占的详细配置：

[root@node10 ~]# cat  /etc/keepalived/keepalived.conf 
! Configuration File for keepalived

global_defs {
  notification_email {
    [e-mail]
    [e-mail]
    [e-mail]
  }
  notification_email_from [e-mail]
  smtp_server 192.168.200.1
  smtp_connect_timeout 30
  router_id LVS_DEVEL
}

vrrp_instance IP1 {
    state BACKUP  #状态必须是BACKUP
    interface eth1
    virtual_router_id 10
    priority 150
    nopreempt #不抢占，优先级比Server 1的同组高
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 5555
    }
    virtual_ipaddress {
        192.168.10.100 
    }
}
vrrp_instance IP2 {
    state BACKUP #状态为BACKUP
    interface eth1
    virtual_router_id 20
    priority 50 #比Server 1的的同组低，就不需要配置不抢占了
    advert_int 1
    authentication {
         auth_type PASS
         auth_pass 1111
    }
    virtual_ipaddress {
         192.168.10.101
     }
}

2.5.3：查看IP现在在那个服务器，将keepalived服务关闭在开启，验证IP地址是否还会配置在本地：

[root@localhost keepalived-1.2.24]# ip addr | grep 192
    inet 192.168.10.128/24 brd 192.168.10.255 scope global br0
    inet 192.168.10.101/32 scope global br0
    inet 192.168.10.100/32 scope global br0
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
[root@localhost keepalived-1.2.24]# /etc/init.d/keepalived   stop
Stopping keepalived (via systemctl):                       [  OK  ]
[root@localhost keepalived-1.2.24]# 
[root@localhost keepalived-1.2.24]# ip addr | grep 192
    inet 192.168.10.128/24 brd 192.168.10.255 scope global br0
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
[root@localhost keepalived-1.2.24]# /etc/init.d/keepalived   start
Starting keepalived (via systemctl):                       [  OK  ]
[root@localhost keepalived-1.2.24]# ip addr | grep 192
    inet 192.168.10.128/24 brd 192.168.10.255 scope global br0
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

2.5.4：验证server 2的IP地址：

[root@node10 ~]# ip addr | grep 192
    inet 192.168.10.129/24 brd 192.168.10.255 scope global eth1
    inet 192.168.10.100/32 scope global eth1
    inet 192.168.10.101/32 scope global eth1
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0

注：由此可见，在非抢占模式下，IP地址发送变动后不会再向优先级高的服务器进行漂移，减少了因为IP变动而引起的网络故障。