一文学会k8s多master集群+keepalived高可用实战

Posted 2023-05-12

参考技术A Apiserver是 kubernetes 集群交互的入口，封装了核心对象的增删改查操作，提供了 RESTFul 风格的 API 接口，通过etcd来实现持久化并维护对象的一致性。所以在整个K8S集群中，Apiserver服务至关重要，一旦宕机，整个K8S平台将无法使用，所以保障企业高可用是运维必备的工作之一。

1.安装keepalived

2.修改配置

3.重启keepalived

Keepalived+HAProxy实现K8S高可用集群-20210927

本文采用Keepalived+HAProxy的方式构建高可用集群。
当你如果你有硬件负载均衡设备当然更好了。

准备环境：

主机	ip
k8s-master01	192.168.10.4
k8s-master02	192.168.10.5
k8s-master03	192.168.10.6
VIP	192.168.10.150

• 架构图
  

注意：master集群采用奇数台数，3、5、7…

• 所有节点都进行hosts文件解析

tail -3 /etc/hosts
192.168.10.4 k8s-master01
192.168.10.5 k8s-master02
192.168.10.6 k8s-master03

• 所有节点都要安装keepalived和haproxy软件

yum -y install haproxy keepalived

• 修改haproxy配置文件（所有节点配置相同）
• 最好选择2.x版本，当然这个版本也不影响使用，只是功能没有2.x版本多

vim /etc/haproxy/haproxy.cfg
global
  maxconn  2000
  ulimit-n  16384
  log  127.0.0.1 local0 err
  stats timeout 30s

defaults
  log global
  mode  http
  option  httplog
  timeout connect 5000
  timeout client  50000
  timeout server  50000
  timeout http-request 15s
  timeout http-keep-alive 15s

frontend monitor-in
  bind *:33305
  mode http
  option httplog
  monitor-uri /monitor

listen stats
  bind    *:8006
  mode    http
  stats   enable
  stats   hide-version
  stats   uri       /stats
  stats   refresh   30s
  stats   realm     Haproxy\\ Statistics
  stats   auth      admin:admin

frontend k8s-master
  bind 0.0.0.0:16443
  bind 127.0.0.1:16443
  mode tcp
  option tcplog
  tcp-request inspect-delay 5s
  default_backend k8s-master

backend k8s-master
  mode tcp
  option tcplog
  option tcp-check
  balance roundrobin
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server k8s-master01	192.168.10.4:6443  check
  server k8s-master02	192.168.10.5:6443  check
  server k8s-master03   192.168.10.6:6443  check

• master01节点修改keepalived配置文件

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3  
    rise 2
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    mcast_src_ip 192.168.10.4
    virtual_router_id 51
    priority 100
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.150/24
    }
    track_script {
       chk_apiserver
    }

• master02节点修改keepalived配置文件

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3  
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    mcast_src_ip 192.168.10.5
    virtual_router_id 51
    priority 50
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.150/24
    }
    track_script {
       chk_apiserver
    }
}

• master03节点修改keepalived配置文件

vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script chk_apiserver {
    script "/etc/keepalived/check_apiserver.sh"
    interval 2
    weight -5
    fall 3  
    rise 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    mcast_src_ip 192.168.10.6
    virtual_router_id 51
    priority 50
    advert_int 2
    authentication {
        auth_type PASS
        auth_pass K8SHA_KA_AUTH
    }
    virtual_ipaddress {
        192.168.10.150/24
    }
    track_script {
       chk_apiserver
    }
}

• 所有节点创建健康检查脚本

vim /etc/keepalived/check_apiserver.sh
#!/bin/bash

err=0
for k in $(seq 1 5)
do
    check_code=$(pgrep haproxy)
    if [[ $check_code == "" ]]; then
        err=$(expr $err + 1)
        sleep 5
        continue
    else
        err=0
        break
    fi
done

if [[ $err != "0" ]]; then
    echo "systemctl stop keepalived"
    /usr/bin/systemctl stop keepalived
    exit 1
else
    exit 0
fi

• 启动haproxy与keepalived服务

systemctl daemon-reload 
systemctl enable --now haproxy
systemctl enable --now keepalived

• 可以用ping和telnet命令测试一下vip的可用性

ping 192.168.10.150
PING 192.168.10.150 (192.168.10.150) 56(84) bytes of data.
64 bytes from 192.168.10.150: icmp_seq=1 ttl=64 time=1.60 ms
64 bytes from 192.168.10.150: icmp_seq=2 ttl=64 time=0.519 ms
64 bytes from 192.168.10.150: icmp_seq=3 ttl=64 time=0.874 ms
64 bytes from 192.168.10.150: icmp_seq=4 ttl=64 time=0.786 ms
^C
--- 192.168.10.150 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 0.519/0.946/1.606/0.403 ms

telnet 192.168.10.150 16443
Trying 192.168.10.150...
Connected to 192.168.10.150.
Escape character is '^]'.
Connection closed by foreign host.

• 再尝试一下断开vip所在节点的keepalived，看ip是否漂移，如果vip漂移至另一节点则代表成功

• 可能难免有地方出错，如果出错可以留言哈