docker-compose之keepalived搭建

Posted 2023-05-24 zhimajiang

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了docker-compose之keepalived搭建相关的知识，希望对你有一定的参考价值。

keepalived的版本如下：

/ # keepalived -v
Keepalived v2.0.20 (01/22,2020)

Copyright(C) 2001-2020 Alexandre Cassen, <acassen@gmail.com>

Built with kernel headers for Linux 4.19.36
Running on Linux 5.10.16.3-microsoft-standard-WSL2 #1 SMP Fri Apr 2 22:23:49 UTC 2021

configure options: --disable-dynamic-linking

Config options:  LIBIPTC LIBIPSET LVS VRRP VRRP_AUTH OLD_CHKSUM_COMPAT FIB_ROUTING

System options:  PIPE2 SIGNALFD INOTIFY_INIT1 VSYSLOG EPOLL_CREATE1 IPV4_DEVCONF IPV6_ADVANCED_API LIBNL3 RTA_ENCAP RTA_EXPIRES RTA_NEWDST RTA_PREF FRA_SUPPRESS_PREFIXLEN FRA_SUPPRESS_IFGROUP FRA_TUN_ID RTAX_CC_ALGO RTAX_QUICKACK RTEXT_FILTER_SKIP_STATS FRA_L3MDEV FRA_UID_RANGE RTAX_FASTOPEN_NO_COOKIE RTA_VIA FRA_OIFNAME FRA_PROTOCOL FRA_IP_PROTO FRA_SPORT_RANGE FRA_DPORT_RANGE RTA_TTL_PROPAGATE IFA_FLAGS IP_MULTICAST_ALL LWTUNNEL_ENCAP_MPLS LWTUNNEL_ENCAP_ILA LIBIPTC NET_LINUX_IF_H_COLLISION NETINET_LINUX_IF_ETHER_H_COLLISION LIBIPVS_NETLINK IPVS_DEST_ATTR_ADDR_FAMILY IPVS_SYNCD_ATTRIBUTES IPVS_64BIT_STATS VRRP_VMAC VRRP_IPVLAN IFLA_LINK_NETNSID CN_PROC SOCK_NONBLOCK SOCK_CLOEXEC O_PATH INET6_ADDR_GEN_MODE VRF SO_MARK SCHED_RESET_ON_FORK

docker-compose.yml文件内容如下：

version: "2.6.1"
services:
  keepalived_master:
    image: osixia/keepalived
    container_name: keepalived_master
    hostname: keepalived_master
    environment:
      TZ: "Asia/Shanghai"
    volumes:
      - ./default1.yaml:/container/environment/99-default/default.yaml
    privileged: true
    networks:
      cmcc_001:
        ipv4_address: "172.21.0.41"
  keepalived_slave_1:
    image: osixia/keepalived
    container_name: keepalived_slave_1
    hostname: keepalived_slave_1
    environment:
      TZ: "Asia/Shanghai"
    volumes:
      - ./default2.yaml:/container/environment/99-default/default.yaml
    privileged: true
    depends_on:
      - keepalived_master
    networks:
      cmcc_001:
        ipv4_address: "172.21.0.42"
  keepalived_slave_2:
    image: osixia/keepalived
    container_name: keepalived_slave_2
    hostname: keepalived_slave_2
    environment:
      TZ: "Asia/Shanghai"
    volumes:
      - ./default3.yaml:/container/environment/99-default/default.yaml
    privileged: true
    depends_on:
      - keepalived_master
    networks:
      cmcc_001:
        ipv4_address: "172.21.0.43"
networks:
  cmcc_001:
    external: true

default1.yaml文件内容如下：

KEEPALIVED_COMMAND_LINE_ARGUMENTS: --log-detail --dump-conf

KEEPALIVED_INTERFACE: eth0
KEEPALIVED_PASSWORD: d0cker

# For electing MASTER, highest priority wins.
# to be MASTER, make 50 more than other machines
KEEPALIVED_PRIORITY: 150

KEEPALIVED_UNICAST_PEERS:
  - 172.21.0.42
  - 172.21.0.43

KEEPALIVED_VIRTUAL_IPS:
  - 192.168.1.231
  - 192.168.1.232

KEEPALIVED_NOTIFY: /container/service/keepalived/assets/notify.sh

KEEPALIVED_ROUTER_ID: 51

KEEPALIVED_STATE: MASTER

default2.yaml文件内容如下：

KEEPALIVED_COMMAND_LINE_ARGUMENTS: --log-detail --dump-conf

KEEPALIVED_INTERFACE: eth0
KEEPALIVED_PASSWORD: d0cker

# For electing MASTER, highest priority wins.
# to be MASTER, make 50 more than other machines
KEEPALIVED_PRIORITY: 130

KEEPALIVED_UNICAST_PEERS:
  - 172.21.0.41
  - 172.21.0.43

KEEPALIVED_VIRTUAL_IPS:
  - 192.168.1.231
  - 192.168.1.232

KEEPALIVED_NOTIFY: /container/service/keepalived/assets/notify.sh

KEEPALIVED_ROUTER_ID: 51

KEEPALIVED_STATE: BACKUP

default3.yaml文件内容如下：

KEEPALIVED_COMMAND_LINE_ARGUMENTS: --log-detail --dump-conf

KEEPALIVED_INTERFACE: eth0
KEEPALIVED_PASSWORD: d0cker

# For electing MASTER, highest priority wins.
# to be MASTER, make 50 more than other machines
KEEPALIVED_PRIORITY: 110

KEEPALIVED_UNICAST_PEERS:
  - 172.21.0.41
  - 172.21.0.42

KEEPALIVED_VIRTUAL_IPS:
  - 192.168.1.231
  - 192.168.1.232

KEEPALIVED_NOTIFY: /container/service/keepalived/assets/notify.sh

KEEPALIVED_ROUTER_ID: 51

KEEPALIVED_STATE: BACKUP

问题1：为什么没有keepalived.conf文件？
答：在使用volumes挂载keepalived.conf文件后，启动容器时会报错。

sed: can\'t move \'/container/service/keepalived/assets/keepalived.confHBCjcp\' to \'/container/service/keepalived/assets/keepalived.conf\': Resource busy

错误的具体描述见链接： https://github.com/osixia/docker-keepalived/issues/5
对错误的具体解释见链接：https://www.cnblogs.com/xuxinkun/p/7116737.html

问题2：如何为不同keepalived容器配置不同的conf文件
答：挂载default.yaml文件到/container/environment/99-default/default.yaml。这个地方还没搞明白，猜测yaml文件是定义变量的，keepalived容器会根据yaml定义的变量，自动生成keepalived.conf文件内容。

原创Linux基础之keepalived

keepalived 2.0.12

官方：http://www.keepalived.org/

一简介

Keepalived is a routing software written in C. The main goal of this project is to provide simple and robust facilities for loadbalancing and high-availability to Linux system and Linux based infrastructures. Loadbalancing framework relies on well-known and widely used Linux Virtual Server (IPVS) kernel module providing Layer4 loadbalancing. Keepalived implements a set of checkers to dynamically and adaptively maintain and manage loadbalanced server pool according their health. On the other hand high-availability is achieved by VRRP protocol. VRRP is a fundamental brick for router failover. In addition, Keepalived implements a set of hooks to the VRRP finite state machine providing low-level and high-speed protocol interactions. In order to offer fastest network failure detection, Keepalived implements BFD protocol. VRRP state transition can take into account BFD hint to drive fast state transition. Keepalived frameworks can be used independently or all together to provide resilient infrastructures.

keepalived是用c写的路由软件，使用vrrp协议（Virtual Router Redundancy Protocol）和arp协议 (Address Resolution Protocol)实现简单和健壮的负载均衡和高可用；

VRRP 将局域网的一组路由器（包括一个Master 即活动路由器和若干个Backup 即备份路由器）组织成一个虚拟路由器，称之为一个备份组。这个虚拟的路由器拥有自己的IP 地址10.100.10.1（这个IP 地址可以和备份组内的某个路由器的接口地址相同，相同的则称为ip拥有者），备份组内的路由器也有自己的IP 地址（如Master的IP 地址为10.100.10.2，Backup 的IP 地址为10.100.10.3）。局域网内的主机仅仅知道这个虚拟路由器的IP 地址10.100.10.1，而并不知道具体的Master 路由器的IP 地址10.100.10.2 以及Backup 路由器的IP 地址10.100.10.3。它们将自己的缺省路由下一跳地址设置为该虚拟路由器的IP 地址10.100.10.1。

原理

主从节点之间通过广播或组播的方式发送vrrp包，然后根据priority来选举出master

14:20:21.521870 IP 192.168.0.1 > 192.168.0.2: VRRPv2, Advertisement, vrid 99, prio 100, authtype simple, intvl 1s, length 20

一旦master一定时间内没有及时发出vrrp包出来，则其他standby会发vrrp包再根据priority选举出master；

master会发送arp包，

Jan 28 19:04:26 cdp-test-server-05 Keepalived_vrrp[27675]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.0.3
Jan 28 19:04:26 cdp-test-server-05 Keepalived_vrrp[27675]: Sending gratuitous ARP on eth0 for 192.168.0.3

注意这里是虚拟ip（vip）的gratuitous ARP，

先看ARP (Address Resolution Protocol, 地址解析协议)，将IP地址转换为MAC地址

ARP的过程：在Host A上发送ARP请求，内容为who has [IP_B], tell [IP_A], 包里携带了主机B的IP地址，以及主机A的IP和MAC。收到广播包的所有主机会检查请求的IP 地址是否是自己的，如果是，就会发送一个ARP应答（单播，从B到A），内容为 [IP_B] is at [MAC_B]，包里携带了主机A和B的MAC及IP地址。

# arping 192.168.0.1

刚才的场景中如果Host A发请求的时候，内容为who has [IP_A], tell [IP_A]，则这是一个gratuitous ARP，为什么会请求自己的IP，因为：正常的ARP是向其他主机请求信息，而免费ARP是主动向其他主机广播自己的信息，所以免费ARP不期待响应；

发送gratuitous ARP后收到广播包的所有主机或者交换机都可以通过命令查看vip和mac（master mac）映射：

# arp -a

这样其他主机就可以通过vip访问到master，也可以通过arp手工绑定

# arp -s 192.168.0.3 00-02-b3-3c-16-95

另外可以通过设置vrrp_garp_master_refresh来让master定期发送gratuitous ARP包；

如果是在云主机环境，就不用考虑搭keepalived了，因为云上通常会禁止vrrp协议的组播以及arp，可以考虑直接用云平台的虚拟ip服务；

二安装

# yum install keepalived

主节点配置

# vi /etc/keepalived/keepalived.conf

vrrp_instance VI_1 {

    state MASTER

    interface eth0

    unicast_src_ip 192.168.0.1

    unicast_peer {

        192.168.0.2

    }

    virtual_router_id 51

    priority 100

    advert_int 1

    authentication {

        auth_type PASS

        auth_pass 1111

    }

    virtual_ipaddress {

        192.168.0.3

    }

}

virtual_server 192.168.0.3 81 {

    delay_loop 6

    lb_algo rr

    lb_kind DR

    nat_mask 255.255.255.0

    persistence_timeout 50

    protocol TCP

    #sorry_server 127.0.0.1 80

    real_server 192.168.0.1 80 {

        weight 1

    }

    real_server 192.168.0.2 80 {

        weight 1

    }

}

这里使用的是单播（unicast_src_ip、unicast_peer ）的方式，因为很多环境下组播不能用，如果想用组播，把单播参数去掉即可；

如果real_server和keealived部署在一台机器上，不需要配置virtual_server；

注释掉

#vrrp_strict

否则会在iptables里生成一条drop规则；

从节点修改配置

state BACKUP
unicast_src_ip 对调
unicast_peer 对调
priority 50

如果开启iptables需要增加规则

# iptables -I INPUT -d 224.0.0.0/8 -j ACCEPT
# iptables -A INPUT -p vrrp -j ACCEPT

启动

# service keepalived start

查看vip

# ip a

查看tcp包

# tcpdump -p vrrp -n

参考：https://docs.oracle.com/cd/E37670_01/E41138/html/section_ksr_psb_nr.html

日志位于/var/log/messages，如果报错：

Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Registering Kernel netlink reflector
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Registering Kernel netlink command channel
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Registering gratuitous ARP shared channel
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Opening file \'/etc/keepalived/keepalived.conf\'.
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: (VI_1): Cannot start in MASTER state if not address owner
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Unable to load ipset library - libipset.so.11: cannot open shared object file: No such file or directory
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: VRRP_Instance(VI_1) removing protocol iptable drop rule
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: Using LinkWatch kernel netlink reflector...
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 27 17:05:05 cdp-test-server-05 Keepalived_vrrp[26508]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]

需要安装ipset

# yum install ipset

然后正常

Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Registering Kernel netlink reflector
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Registering Kernel netlink command channel
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Registering gratuitous ARP shared channel
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Opening file \'/etc/keepalived/keepalived.conf\'.
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: (VI_1): Cannot start in MASTER state if not address owner
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: iptc_commit returned 0: No chain/target/match by that name
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) removing protocol VIPs.
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) removing protocol iptable drop rule
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: Using LinkWatch kernel netlink reflector...
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Entering BACKUP STATE
Jan 27 21:40:35 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]
Jan 27 21:40:38 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) setting protocol iptable drop rule
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:39 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: VRRP_Instance(VI_1) Sending/queueing gratuitous ARPs on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3
Jan 27 21:40:44 cdp-test-server-05 Keepalived_vrrp[7001]: Sending gratuitous ARP on eth0 for 192.168.0.3