Kubernetes高可用集群二进制部署

Posted 123坤

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了Kubernetes高可用集群二进制部署相关的知识,希望对你有一定的参考价值。

Kubernetes高可用集群二进制部署

Kubernetes高可用集群二进制部署

Kubernetes(简称为:k8s)是Google在2014年6月开源的一个容器集群管理系统,使用Go语言开发,用于管理云平台中多个主机上的容器化的应用,Kubernetes的目标是让部署容器化的应用简单并且高效,Kubernetes提供了资源调度、部署管理、服务发现、扩容缩容、监控,维护等一整套功能,努力成为跨主机集群的自动部署、扩展以及运行应用程序容器的平台。 它支持一系列容器工具, 包括Docker、Containerd等。

一、集群环境准备

1.1 主机规划

主机IP地址主机名主机配置主机角色软件列表
192.168.10.101k8s-master12C4Gmasterkube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce
192.168.10.102k8s-master22C4Gmasterkube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce
192.168.10.103k8s-master32C4Gmasterkube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce
192.168.10.104k8s-worker022C4Gworkerkubelet、kube-proxy、docker-ce
192.168.10.105ha11C2GLBhaproxy、keepalived
192.168.10.106ha21C2GLBhaproxy、keepalived
192.168.10.100//VIP(虚拟IP)

1.2 软件版本

软件名称版本备注
CentOS7kernel版本:5.16
kubernetesv1.21.10
etcdv3.5.2最新版本
calicov3.19.4
corednsv1.8.4

1.3 网络分配

网络名称网段备注
Node网络192.168.10.0/24
Service网络10.96.0.0/16
Pod网络10.244.0.0/16

二、集群部署

2.1主机准备

2.1.1 主机名设置

修改主机名为上述主机规划中的名字

hostnamectl set-hostname xxx

2.1.2 主机与IP地址解析

cat >> /etc/hosts << EOF
192.168.10.101 k8s-master01 
192.168.10.102 k8s-master02
192.168.10.103 k8s-master03 k8s-worker01
192.168.10.104 k8s-worker02
192.168.10.105 ha1
192.168.10.106 ha2
EOF

2.1.3 主机安全设置

2.1.3.1 关闭防火墙

systemctl stop firewalld
systemctl disable firewalld
firewall-cmd --state

2.1.3.2 关闭selinux

setenforce 0
sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
sestatus

2.1.4 交换分区设置

swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
echo "vm.swappiness=0" >> /etc/sysctl.conf
sysctl -p

2.1.5 主机系统时间同步

安装软件
yum -y install ntpdate

制定时间同步计划任务
crontab -e
0 */1 * * * ntpdate time1.aliyun.com

2.1.6 主机系统优化

ulimit -SHn 65535
cat <<EOF >> /etc/security/limits.conf
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF

2.1.7 ipvs管理工具安装及模块加载

为集群节点安装,负载均衡节点不用安装

yum -y install ipvsadm ipset sysstat conntrack libseccomp
创建 /etc/modules-load.d/ipvs.conf 并加入以下内容: 
cat >/etc/modules-load.d/ipvs.conf <<EOF 
ip_vs 
ip_vs_lc 
ip_vs_wlc 
ip_vs_rr 
ip_vs_wrr 
ip_vs_lblc 
ip_vs_lblcr 
ip_vs_dh 
ip_vs_sh 
ip_vs_fo 
ip_vs_nq 
ip_vs_sed 
ip_vs_ftp 
ip_vs_sh 
nf_conntrack 
ip_tables 
ip_set 
xt_set 
ipt_set 
ipt_rpfilter 
ipt_REJECT 
ipip 
EOF
设置为开机启动
systemctl enable --now systemd-modules-load.service

2.1.8 Linux内核升级

在所有节点中安装,需要重新操作系统更换内核。

[root@localhost ~]# yum -y install perl
[root@localhost ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[root@localhost ~]# yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
[root@localhost ~]# yum  --enablerepo="elrepo-kernel"  -y install kernel-ml.x86_64
[root@localhost ~]# grub2-set-default 0
[root@localhost ~]# grub2-mkconfig -o /boot/grub2/grub.cfg

2.1.9 Linux内核优化

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 131072
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system
所有节点配置完内核后,重启服务器,保证重启后内核依旧加载
reboot -h now
重启后查看结果:
lsmod | grep --color=auto -e ip_vs -e nf_conntrack

2.1.10 其它工具安装(选装)

yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz -y

2.2 负载均衡器准备

2.2.1 安装haproxy与keepalived

在负载均衡节点安装对应的包

yum -y install haproxy keepalived

2.2.2 HAProxy配置

cat >/etc/haproxy/haproxy.cfg<<"EOF"
global
 maxconn 2000
 ulimit-n 16384
 log 127.0.0.1 local0 err
 stats timeout 30s

defaults
 log global
 mode http
 option httplog
 timeout connect 5000
 timeout client 50000
 timeout server 50000
 timeout http-request 15s
 timeout http-keep-alive 15s

frontend monitor-in
 bind *:33305
 mode http
 option httplog
 monitor-uri /monitor

frontend k8s-master
 bind 0.0.0.0:6443
 bind 127.0.0.1:6443
 mode tcp
 option tcplog
 tcp-request inspect-delay 5s
 default_backend k8s-master

backend k8s-master
 mode tcp
 option tcplog
 option tcp-check
 balance roundrobin
 default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
 server  k8s-master01  192.168.10.101:6443 check
 server  k8s-master02  192.168.10.102:6443 check
 server  k8s-master03  192.168.10.103:6443 check
EOF

2.2.3 KeepAlived

主从配置不一致,需要注意。

ha1:

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs 
   router_id LVS_DEVEL
script_user root
   enable_script_security

vrrp_script chk_apiserver 
   script "/etc/keepalived/check_apiserver.sh"
   interval 5
   weight -5
   fall 2 
rise 1

vrrp_instance VI_1 
   state MASTER
   interface ens33
   mcast_src_ip 192.168.10.105
   virtual_router_id 51
   priority 100
   advert_int 2
   authentication 
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   
   virtual_ipaddress 
       192.168.10.100
   
   track_script 
      chk_apiserver
   

EOF
ha2:

cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs 
   router_id LVS_DEVEL
script_user root
   enable_script_security

vrrp_script chk_apiserver 
   script "/etc/keepalived/check_apiserver.sh"
  interval 5
   weight -5
   fall 2 
rise 1

vrrp_instance VI_1 
   state BACKUP
   interface ens33
   mcast_src_ip 192.168.10.106
   virtual_router_id 51
   priority 99
   advert_int 2
   authentication 
       auth_type PASS
       auth_pass K8SHA_KA_AUTH
   
   virtual_ipaddress 
       192.168.10.100
   
   track_script 
      chk_apiserver
   

EOF

2.2.4 健康检查脚本

ha1及ha2均要配置

cat > /etc/keepalived/check_apiserver.sh <<"EOF"
#!/bin/bash
err=0
for k in $(seq 1 3)
do
   check_code=$(pgrep haproxy)
   if [[ $check_code == "" ]]; then
       err=$(expr $err + 1)
       sleep 1
       continue
   else
       err=0
       break
   fi
done

if [[ $err != "0" ]]; then
   echo "systemctl stop keepalived"
   /usr/bin/systemctl stop keepalived
   exit 1
else
   exit 0
fi
EOF
chmod +x /etc/keepalived/check_apiserver.sh

2.2.5 启动服务并验证

systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived
ip address show

2.3 配置免密登录

在k8s-master01上操作

ssh-keygen
ssh-copy-id root@k8s-master01
ssh-copy-id root@k8s-master02
ssh-copy-id root@k8s-master03
ssh-copy-id root@k8s-worker01
ssh root@k8s-master01

2.4 部署ETCD集群

在k8s-master01上操作。

2.4.1 创建工作目录

mkdir -p /data/k8s-work

2.4.2 获取cfssl工具

cd /data/k8s-work
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
说明:
cfssl是使用go编写,由CloudFlare开源的一款PKI/TLS工具。主要程序有:
- cfssl,是CFSSL的命令行工具
- cfssljson用来从cfssl程序获取JSON输出,并将证书,密钥,CSR和bundle写入文件中。
chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
# cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6

2.4.3 创建CA证书

2.4.3.1 配置ca证书请求文件

cat > ca-csr.json <<"EOF"

  "CN": "kubernetes",
  "key": 
      "algo": "rsa",
      "size": 2048
  ,
  "names": [
    
      "C": "CN",
      "ST": "Guangdong",
      "L": "Shenzhen",
      "O": "kubemsb",
      "OU": "CN"
    
  ],
  "ca": 
          "expiry": "87600h"
  

EOF

2.4.3.2 创建ca证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

2.4.3.3 配置ca证书策略

cat > ca-config.json <<"EOF"

  "signing": 
      "default": 
          "expiry": "87600h"
        ,
      "profiles": 
          "kubernetes": 
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          
      
  

EOF
server auth 表示client可以对使用该ca对server提供的证书进行验证
client auth 表示server可以使用该ca对client提供的证书进行验证

2.4.4 创建etcd证书

2.4.4.1 配置etcd请求文件

cat > etcd-csr.json <<"EOF"

  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.10.101",
    "192.168.10.102",
    "192.168.10.103"
  ],
  "key": 
    "algo": "rsa",
    "size": 2048
  ,
  "names": [
    "C": "CN",
    "ST": "Guangdong",
    "L": "Shenzhen",
    "O": "kubemsb",
    "OU": "CN"
  ]

EOF

2.4.4.2 生成etcd证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson  -bare etcd
# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  etcd.csr  etcd-csr.json  etcd-key.pem  etcd.pem

2.4.5 部署etcd集群

2.4.5.1 下载etcd软件包

wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz

2.4.5.2 安装etcd软件

tar -xvf etcd-v3.5.2-linux-amd64.tar.gz
cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/

2.4.5.3 分发etcd软件

scp etcd-v3.5.2-linux-amd64/etcd* k8s-master02:/usr/local/bin/
scp etcd-v3.5.2-linux-amd64/etcd* k8s-master03:/usr/local/bin/

2.4.5.4 创建配置文件

mkdir /etc/etcd
cat >  /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.101:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.101:2379,http://127.0.0.1:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.101:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.101:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.101:2380,etcd2=https://192.168.10.102:2380,etcd3=https://192.168.10.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
说明:
ETCD_NAME:节点名称,集群中唯一
ETCD_DATA_DIR:数据目录
ETCD_LISTEN_PEER_URLS:集群通信监听地址
ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
ETCD_INITIAL_CLUSTER:集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN:集群Token
ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群

2.4.5.5 创建服务配置文件

mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd
cd /data/k8s-work
cp ca*.pem /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl
cat > /etc/systemd/system/etcd.service <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \\
  --cert-file=/etc/etcd/ssl/etcd.pem \\
  --key-file=/etc/etcd/ssl/etcd-key.pem \\
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \\
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
EOF

2.4.5.6 同步etcd配置到集群其它master节点

创建目录
mkdir -p /etc/etcd
mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd
服务配置文件,需要修改etcd节点名称及IP地址
for i in k8s-master02 k8s-master03 \\
do \\
scp /etc/etcd/etcd.conf $i:/etc/etcd/ \\
done
k8s-master02:
cat /etc/etcd/etcd.conf
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.102:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.102:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.102:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.102:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.101:2380,etcd2=https://192.168.10.102:2380,etcd3=https://192.168.10.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
k8s-master03:
cat /etc/etcd/etcd.conf
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.10.103:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.10.103:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.10.103:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.10.103:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.10.101:2380,etcd2=https://192.168.10.102:2380,etcd3=https://192.168.10.103:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
证书文件
for i in k8s-master2 k8s-master3 \\
do \\
scp /etc/etcd/ssl/* $i:/etc/etcd/ssl \\
done
服务启动配置文件
for i in k8s-master2 k8s-master3 \\
do \\
scp /etc/systemd/system/etcd.service $i:/etc/systemd/system/ \\
done

2.4.5.7 启动etcd集群

systemctl daemon-reload
systemctl enable --now etcd.service
systemctl status etcd

2.4.5.8 验证集群状态

ETCDCTL_API=3 /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.101:2379,https://192.168.10.102:2379,https://192.168.10.103:2379 endpoint health
+----------------------------+--------+-------------+-------+
|          ENDPOINT          | HEALTH |    TOOK     | ERROR |
+----------------------------+--------+-------------+-------+
| https://192.168.10.101:2379 |   true | 10.393062ms |       |
| https://192.168.10.102:2379 |   true |  15.70437ms |       |
| https://192.168.10.103:2379 |   true | 15.871684ms |       |
+----------------------------+--------+-------------+-------+

2.5 Kubernetes集群部署

2.5.1 Kubernetes软件包下载

wget https://dl.k8s.io/v1.21.10/kubernetes-server-linux-amd64.tar.gz

2.5.2 Kubernetes软件包安装

tar -xvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/
cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/

2.5.3 Kubernetes软件分发

scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master02:/usr/local/bin/
scp kube-apiserver kube-controller-manager kube-scheduler kubectl k8s-master03:/usr/local/bin/
scp kubelet kube-proxy k8s-master01:/usr/local/bin
scp kubelet kube-proxy k8s-master02:/usr/local/bin
scp kubelet kube-proxy k8s-master03:/usr/local/bin
scp kubelet kube-proxy k8s-worker01:/usr/local/bin

2.5.4 在集群节点上创建目录

所有节点

mkdir -p /etc/kubernetes/        
mkdir -p /etc/kubernetes/ssl     
mkdir -p /var/log/kubernetes 

2.5.5 部署api-server

2.5.5.1 创建apiserver证书请求文件

cat > kube-apiserver-csr.json << "EOF"

"CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.10.101",
    "192.168.10.102",
    "192.168.10.103",
    "192.168.10.104",
    "192.168.10.105",
    "192.168.10.106",
    "192.168.10.107",
    "192.168.10.108",
    "192.168.10.109",
    "192.168.10.110",
    "192.168.10.100",
    "10.96.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": 
    "algo": "rsa",
    "size": 2048
  ,
  "names": [
    
      "C": "CN",
      "ST": "Guangdong",
      以上是关于Kubernetes高可用集群二进制部署的主要内容,如果未能解决你的问题,请参考以下文章

2.还不会部署高可用的kubernetes集群?看我手把手教你使用二进制部署v1.23.6的K8S集群实践(下)

使用二进制的方式部署 K8S-1.16 高可用集群(二十)

快速搭建kubernetes高可用集群(3master+3worker+负载均衡)

K8s二进制安装(k8s1.17.4集群+keepalive-haproxy高可用)

高可用部署二进制 Kubernetes

如何在Kubernetes中部署一个高可用的PostgreSQL集群环境