k8s/kubeadm 生产环境高可用集群部署
Posted 不用去猜
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了k8s/kubeadm 生产环境高可用集群部署相关的知识，希望对你有一定的参考价值。
基本环境配置
kubeadm 安装方式自 1.14 版本以后，安装方法几乎没有任何变化，此文档可以尝试安装最新的 k8s 集群， centos 采用的是 7.x 版本
k8s 官网： https://kubernetes.io/docs/setup
最新版高可用安装 ： https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
高可用 kubernetes 集群规划
| 主机名 | IP地址 | 说明 |
|---|---|---|
| k8s-master01 ~ 03 | 192.168.32.129 ~ 131 | master 节点 ；3个 |
| k8s-master-lb | 192.168.32.233 | keepalived 虚拟 IP |
| node01 ~ 02 | 192.168.32.132 ~ 133 | worker 节点 ；2个 |
| 配置信息 | 备注 |
|---|---|
| 系统版本 | Centos7.x |
| Docker 版本 | 19.03.x |
| Pod 网段 | 172.168.0.0/12 |
| Service 网段 | 10.96.0.0/12 |
VIP (虚拟IP) 不要和公司内网 IP 重复， 首先去 ping 一下，不通才可用。VIP 需要和主机在同一个局域网内！
所有节点配置 hosts, 修改 /etc/hosts 文件 如下：
[root@k8s-master01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.32.129 k8s-master01
192.168.32.130 k8s-master02
192.168.32.131 k8s-master03
192.168.32.132 node01
192.168.32.133 node02
Centos 7 安装 yum 源如下；
yum install -y wget
wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo #阿里yum base源
wget -P /etc/yum.repos.d/ http://mirrors.aliyun.com/repo/epel-7.repo #阿里 epel 源
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
sed -i -e \'/mirrors.cloud.aliyuncs.com/d\' -e \'/mirrors.aliyuncs.com/d\' /etc/yum.repos.d/CentOS-Base.repo
必备工具安装
yum -y install wget jq psmics vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git
所有节点关闭防火墙，selinux, dnsmasq , swap
systemctl disable --now firewalld
systemctl disable --now dnsmasq
systemctl disable --now NetworkManager
setenforce 0
sed -i \'s#SELINUX=enforcing#SELINUX=disabled#g\' /etc/sysconfig/selinux
sed -i \'s#SELINUX=enforcing#SELINUX=disabled#g\' /etc/selinux/config
关闭 swap 分区
swapoff -a && sysctl -w vm.swappiness=0
sed -ri \'/^[^#]*swap/s@^@#@\' /etc/fstab
安装 ntpdate
rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
yum -y install ntpdate
所有节点同步时间，时间同步配置如下
ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
echo \'Asia/Shanghai\' > /etc/timezone
ntpdate time2.aliyun.com
加入到 crontab
[root@k8s-master01 yum.repos.d]# crontab -l
*/5 * * * * /usr/sbin/ntpdate time2.aliyun.com
所有节点配置 limit
ulimit -SHn 65535
vim /etc/security/limits.conf
# 末尾添加如下内容
[root@k8s-master01 yum.repos.d]# tail -6 /etc/security/limits.conf
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
[root@k8s-master01 yum.repos.d]#
Master01 节点免密钥登陆其他节点，安装过程中生成配置文件和证书（kubeadm不需要手动生成证书)均在 Master01 上操作，集群管理也在 Master01 上操作，阿里云或者 AWS 上需要单独一台 kubectl 服务器。密钥配置如下；
ssh-keygen -t rsa
for i in k8s-master01 k8s-master02 k8s-master03 node01 node02;do ssh-copy-id -i /root/.ssh/id_rsa.pub $i;done
下载安装所有的源码文件
git clone https://gitee.com/dukuan/k8s-ha-install.git
所有节点升级系统并重启，此处升级没有升级内核，下面会单独升级内核
yum update -y --exclude=kernel* && reboot #CentOs7 需要升级
内核配置
Centos7 需要升级内核至4.18+，本次升级的版本为4.19
在 master01 节点下载内核
cd /root
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm
wget http://193.49.22.109/elrepo/kernel/el7/x86_64/RPMS/kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
for i in k8s-master01 k8s-master02 k8s-master03 node01 node02;do scp kernel-ml-devel-4.19.12-1.el7.elrepo.x86_64.rpm kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm $i:/root/;done #分发到其他节点
所有节点升级内核
yum localinstall -y kernel-ml*
所有节点更改内核启动顺序
grub2-set-default 0 && grub2-mkconfig -o /etc/grub2.cfg
grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)"
检查默认内核是不是4.19
[root@k8s-master01 ~]# grubby --default-kernel
/boot/vmlinuz-4.19.12-1.el7.elrepo.x86_64
所有节点重启，然后检查内核是不是 4.19
[root@k8s-master01 ~]# uname -a
Linux k8s-master01 4.19.12-1.el7.elrepo.x86_64 #1 SMP Fri Dec 21 11:06:36 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
所有节点安装 ipvsadm
yum install ipvsadm ipset sysstat conntrack libseccomp -y
所有节点配置 ipvs 模块， 在内核 4.19+ 版本 nf_conntrack_ipv4 已经改为 nf_conntrack, 4.18 以下使用 nf_conntrack_ipv4 即可
vim /etc/modules-load.d/ipvs.conf #默认不存在
[root@k8s-master01 ~]# cat /etc/modules-load.d/ipvs.conf #加入如下配置
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
然后执行 systemctl enable --now systemd-modules-load.service 即可
开启一些k8s集群中必须的内核参数，所有节点配置k8s内核
[root@k8s-master01 ~]# cat <<EOF > /etc/sysctl.d/k8s.conf
> net.ipv4.ip_forward = 1
> net.bridge.bridge-nf-call-iptables = 1
> net.bridge.bridge-nf-call-ip6tables = 1
> fs.may_detach_mounts = 1
> net.ipv4.conf.all.route_localnet = 1
> vm.overcommit_memory=1
> vm.panic_on_oom=0
> fs.inotify.max_user_watches=89100
> fs.file-max=52706963
> fs.nr_open=52706963
> net.netfilter.nf_conntrack_max=2310720
> net.ipv4.tcp_keepalive_time = 600
> net.ipv4.tcp_keepalive_probes = 3
> net.ipv4.tcp_keepalive_intvl =15
> net.ipv4.tcp_max_tw_buckets = 36000
> net.ipv4.tcp_tw_reuse = 1
> net.ipv4.tcp_max_orphans = 327680
> net.ipv4.tcp_orphan_retries = 3
> net.ipv4.tcp_syncookies = 1
> net.ipv4.tcp_max_syn_backlog = 16384
> net.ipv4.ip_conntrack_max = 65536
> net.ipv4.tcp_max_syn_backlog = 16384
> net.ipv4.tcp_timestamps = 0
> net.core.somaxconn = 16384
> EOF
sysctl --system
所有节点配置完内核后，重启服务器，保证重启后内核加载
reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack
基本组件安装
docker-ce, kubernetes 各组件等
所有节点安装 docker-ce 19.03
yum install docker-ce-19.03.* -y
温馨提示；
由于新版 kubelet 建议使用 systemd , 所以可以把 docker 的 CgroupDriver 改成 systemd
mkdir /etc/docker
cat > /etc/docker/daemon.json <<EOF
"registry-mirrors": ["https://6h6ezoe5.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
EOF
所有节点设置开机自启动 docker
systemctl daemon-reload && systemctl enable --now docker
安装 k8s 组件
yum list kubeadm.x86_64 --showduplicates | sort -r #查看版本
所有节点安装最新版本kubeadm
yum -y install kubeadm 安装最新版
yum -y install kubeadm-1.22.5-0 指定版本
ps: 由于官网未开放同步方式, 可能会有索引gpg检查失败的情况, 这时请用
yum -y install kubeadm-1.22.5-0 --nogpgcheck
默认配置的 pause 镜像使用 gcr.io 仓库，国内可能无法访问，所以这里配置 kubelet 使用阿里云的 pause 镜像
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.2"
EOF
设置 kubelet 开机自启动
systemctl daemon-reload
systemctl enable --now kubelet #这时候查看kubelet 启动状态是失败的，没有关系，因为缺少配置
高可用组件安装
（注意： 如果不是高可用集群， haproxy 和 keepalived 无需安装 )
公有云要用公有云自带的负载均衡，比如阿里云的SLB，腾讯云的ELB，用来替代haproxy和keepalived，因为公有云大部分都是不支持keepalived的，另外如果用阿里云的话，kubectl控制端不能放在master节点，推荐使用腾讯云，因为阿里云的slb有回环的问题，也就是slb代理的服务器不能反向访问SLB，但是腾讯云修复了这个问题。
所有 Master 节点通过 yum 安装 HAproxy 和 KeepAlived:
yum -y install keepalived haproxy
所有 Master 节点配置 HAproxy (所有 Master 节点的 HAProxy配置相同)
[root@k8s-master01 ~]# cat /etc/haproxy/haproxy.cfg
global
maxconn 2000
ulimit-n 16384
log 127.0.0.1 local0 err
stats timeout 30s
defaults
log global
mode http
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000
timeout http-request 15s
timeout http-keep-alive 15s
frontend monitor-in
bind *:33305
mode http
option httplog
monitor-uri /monitor
frontend k8s-master
bind 0.0.0.0:16443
bind 127.0.0.1:16443
mode tcp
option tcplog
tcp-request inspect-delay 5s
default_backend k8s-master
backend k8s-master
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server k8s-master01 192.168.32.129:6443 check
server k8s-master02 192.168.32.130:6443 check
server k8s-master03 192.168.32.131:6443 check
[root@k8s-master01 ~]#
注意：master 的ip地址
所有 Master 节点配置 keepalived,配置不一样，注意区分
每台服务器 优先级必须不同 priority 100 其他机器设置为 99 98
master01 配置：
[root@k8s-master01 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs
router_id LVS_DEVEL
script_user root
enable_script_security
vrrp_script chk_apiserver
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
vrrp_instance VI_1
state MASTER
interface ens32
mcast_src_ip 192.168.32.129
virtual_router_id 51
priority 101
advert_int 2
authentication
auth_type PASS
auth_pass K8SHA_KA_AUTH
virtual_ipaddress
192.168.32.233
track_script
chk_apiserver
[root@k8s-master01 ~]#
Master02 配置：
[root@k8s-master02 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs
router_id LVS_DEVEL
script_user root
enable_script_security
vrrp_script chk_apiserver
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
vrrp_instance VI_1
state MASTER
interface ens32
mcast_src_ip 192.168.32.130
virtual_router_id 51
priority 90
advert_int 2
authentication
auth_type PASS
auth_pass K8SHA_KA_AUTH
virtual_ipaddress
192.168.32.233
track_script
chk_apiserver
[root@k8s-master02 ~]#
Master03 配置
[root@k8s-master03 ~]# cat /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs
router_id LVS_DEVEL
script_user root
enable_script_security
vrrp_script chk_apiserver
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
vrrp_instance VI_1
state MASTER
interface ens32
mcast_src_ip 192.168.32.131
virtual_router_id 51
priority 80
advert_int 2
authentication
auth_type PASS
auth_pass K8SHA_KA_AUTH
virtual_ipaddress
192.168.32.233
track_script
chk_apiserver
You have new mail in /var/spool/mail/root
[root@k8s-master03 ~]#
健康检查配置 （所有master 节点添加健康检查脚本）
cat > /etc/keepalived/check_apiserver.sh <<"EOF"
#!/bin/bash
err=0
for k in $(seq 1 3)
do
check_code=$(pgrep haproxy)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_apiserver.sh
所有 master 节点启动 haproxy 和 keepalived
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived
测试 haproxy 与 keepalived 是否正常
重要：如果安装了keepalived和haproxy，需要测试keepalived是否是正常的
所以这里需要测试VIP是否通
[root@k8s-master01 ~]# ip addr show ens32
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:ce:bd:c6 brd ff:ff:ff:ff:ff:ff
inet 192.168.32.129/24 brd 192.168.32.255 scope global dynamic ens32
valid_lft 1745sec preferred_lft 1745sec
inet 192.168.32.233/32 scope global ens32
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fece:bdc6/64 scope link
valid_lft forever preferred_lft forever
[root@k8s-master01 ~]#
可以看到master01 节点 虚拟ip 已经起来了，接下来 Ping 一下 看看是否联通
[root@k8s-master01 ~]# ping 192.168.32.233
PING 192.168.32.233 (192.168.32.233) 56(84) bytes of data.
64 bytes from 192.168.32.233: icmp_seq=1 ttl=64 time=0.083 ms
64 bytes from 192.168.32.233: icmp_seq=2 ttl=64 time=0.026 ms
64 bytes from 192.168.32.233: icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from 192.168.32.233: icmp_seq=4 ttl=64 time=0.126 ms
^C
telnet测试：
[root@k8s-master01 ~]# telnet 192.168.32.233 16443
Trying 192.168.32.233...
Connected to 192.168.32.233.
Escape character is \'^]\'.
Connection closed by foreign host.
注意： 如果ping不通且telnet没有出现 ] ，则认为VIP不可以，不可在继续往下执行，需要排查keepalived的问题，比如防火墙和selinux，haproxy和keepalived的状态，监听端口等
所有节点查看防火墙状态必须为disable和inactive：systemctl status firewalld
所有节点查看selinux状态，必须为disable：getenforce
master节点查看haproxy和keepalived状态：systemctl status keepalived haproxy
master节点查看监听端口：netstat -lntp
kubernetes 集群初始化
官方初始化文档： https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
Master01节点创建 kubeadm-config.yaml 配置文件如下：
Master01：（# 注意，如果不是高可用集群，192.168.32.233:16443改为master01的地址，16443改为apiserver的端口，默认是6443，注意更改v1.18.5自己服务器kubeadm的版本：kubeadm version）
查看办法：
kubectl version
[root@k8s-master01 ~]# kubectl version
Client Version: version.InfoMajor:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"
因为安装的版本是GitVersion:"v1.23.1"
下面的 yaml 文件中的对应版本需改为 v1.23.1
cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: 7t2weq.bjbawausm0jaxury
ttl: 24h0m0s #token 过期时间
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.32.129
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
imagePullPolicy: IfNotPresent
name: k8s-master01
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
certSANs:
- 192.168.32.233
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.32.233:16443
controllerManager:
dns:
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.23.1
networking:
dnsDomain: cluster.local
podSubnet: 172.168.0.0/12
serviceSubnet: 10.96.0.0/12
scheduler:
更新kubeadm文件
kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml
将new.yaml文件复制到其他master节点，之后所有Master节点提前下载镜像，可以节省初始化时间： for i in k8s-master02 k8s-master03; do scp new.yaml $i:/root/; done
kubeadm config images pull --config /root/new.yaml
所有节点设置开机自启动kubelet systemctl daemon-reload systemctl enable --now kubelet
systemctl daemon-reload
systemctl enable --now kubelet
master01初始化
Master01节点初始化，初始化以后会在/etc/kubernetes目录下生成对应的证书和配置文件，之后其他Master节点加入Master01即可：
#master01节点执行初始化：
kubeadm init --config /root/new.yaml --upload-certs
#如果初始化失败，重置后再次初始化，命令如下：
kubeadm reset -f ; ipvsadm --clear ; rm -rf ~/.kube
初始化关键信息，记录一下，会用到
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of the control-plane node running the following command on each as root:
kubeadm join 192.168.32.233:16443 --token 7t2weq.bjbawausm0jaxury \\
--discovery-token-ca-cert-hash sha256:2aec4a48ad70c0241a103168f69a3bdb60c75f71fc98fa8188489632288683ea \\
--control-plane --certificate-key 1a02bbeefc83fc2fa9b313a955fcfec42680fc5aeb2ec4cb2f49747fdf8f78ed
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.32.233:16443 --token 7t2weq.bjbawausm0jaxury \\
--discovery-token-ca-cert-hash sha256:2aec4a48ad70c0241a103168f69a3bdb60c75f71fc98fa8188489632288683ea
Master01 节点配置环境变量，用于访问 kubernetes 集群
[root@k8s-master01 ~]# cat <<EOF >> /root/.bashrc
> export KUBECONFIG=/etc/kubernetes/admin.conf
> EOF
source /root/.bashrc
master02 加入集群
[root@k8s-master02 ~]# kubeadm join 192.168.32.233:16443 --token 7t2weq.bjbawausm0jaxury --discovery-token-ca-cert-hash sha256:2aec4a48ad70c0241a103168f69a3bdb60c75f71fc98fa8188489632288683ea --control-plane --certificate-key 1a02bbeefc83fc2fa9b313a955fcfec42680fc5aeb2ec4cb2f49747fdf8f78ed
模拟token 过期后生成新的 token #token 过期很常见，比如扩容集群的时候
[root@k8s-master01 ~]# kubeadm token create --print-join-command
kubeadm join 192.168.32.233:16443 --token wqhr43.i9dukjwmu7bti1ly --discovery-token-ca-cert-hash sha256:2aec4a48ad70c0241a103168f69a3bdb60c75f71fc98fa8188489632288683ea
node 节点加入集群，直接输入上面那段就可以了
[root@k8s-master01 ~]# kubeadm init phase upload-certs --upload-certs
I0118 19:40:21.507779 52890 version.go:255] remote version is much newer: v1.23.1; falling back to: stable-1.22
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
b226aba32c0a5d618f74771deec1a9a19a5a77f332ebc61fb930328c3f6b6d52 #key
master 节点加入集群
kubeadm join 192.168.32.233:16443 --token wqhr43.i9dukjwmu7bti1ly --discovery-token-ca-cert-hash sha256:2aec4a48ad70c0241a103168f69a3bdb60c75f71fc98fa8188489632288683ea \\
--control-plane --certificate-key \\
b226aba32c0a5d618f74771deec1a9a19a5a77f332ebc61fb930328c3f6b6d52
Calico 网络组件的安装
以下步骤只在 master01 执行
cd /root/k8s-ha-install/ && git checkout manual-installation-v1.21.x && cd calico
修改 calico-etcd.yaml 文件的以下位置
sed -i \'s#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.32.129:2379,https://192.168.32.130:2379,https://192.168.32.131:2379"#g\' calico-etcd.yaml
ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d \'\\n\'`
ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d \'\\n\'`
ETCD_KEY=`cat /etc/kubernetes/pki/etcd/server.key |base64 | tr -d \'\\n\'`
sed -i "s@# etcd-key: null@etcd-key: $ETCD_KEY@g; s@# etcd-cert: null@etcd-cert: $ETCD_CERT@g; s@# etcd-ca: null@etcd-ca: $ETCD_CA@g" calico-etcd.yaml
sed -i \'s#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g\' calico-etcd.yaml
POD_SUBNET=`cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= \'print $NF\'`
sed -i \'s@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@# value: "192.168.0.0/16"@ value: \'"$POD_SUBNET"\'@g\' calico-etcd.yaml
声明式创建资源
kubectl apply -f calico-etcd.yaml
Metrics 部署
在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率
#将Master01节点的front-proxy-ca.crt复制到所有Node节点
scp -r /etc/kubernetes/pki/front-proxy-ca.crt node01:/etc/kubernetes/pki/front-proxy-ca.crt
scp -r /etc/kubernetes/pki/front-proxy-ca.crt node02:/etc/kubernetes/pki/front-proxy-ca.crt
scp -r /etc/kubernetes/pki/front-proxy-ca.crt k8s-master02:/etc/kubernetes/pki/front-proxy-ca.crt
scp -r /etc/kubernetes/pki/front-proxy-ca.crt k8s-master03:/etc/kubernetes/pki/front-proxy-ca.crt
#安装Metrics
cd /root/k8s-ha-install/metrics-server-0.4.x-kubeadm/
kubectl create -f comp.yaml
安装dashboard
cd /root/k8s-ha-install/dashboard/
kubectl create -f .
查看 svc 的 nodeport
[root@k8s-master01 ~/k8s-ha-install/dashboard]# kubectl get svc -n kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
dashboard-metrics-scraper ClusterIP 10.107.249.230 <none> 8000/TCP 16m
kubernetes-dashboard NodePort 10.109.229.183 <none> 443:31195/TCP 16m
浏览器访问 https://192.168.32.233:31195
获取token
[root@k8s-master01 ~/k8s-ha-install/dashboard]# kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk \'print $1\')
Name: admin-user-token-vnfd6
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: 80d1b524-586e-4818-83db-521393d4f7c0
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkVCN21ZUWxYenphNmlnODFSaUlsOUtRWGQyUFQweWhfS2k3YlhCWDRXYm8ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXZuZmQ2Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiI4MGQxYjUyNC01ODZlLTQ4MTgtODNkYi01MjEzOTNkNGY3YzAiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.OiFN1Kd1fOHT3ExWIodZf6Nee-s5zWq_rMwHadzHKoaYLx9h3da1kY4qNI6F5Zcz5-HGj4IByZpcoldv_J00JaXLl8Js5fFoELe-emVpIX10CFnBaBs5GuGa-khlwgTInxKEol-8089iaRCDM4imI1HCrNF4btkcL96e5eoAUBVIiiSNWI7sxEphAlL4Dg7LGLmNdb7AJZaMzdPWori6F7CfzZ_OnueFYPgtPaboKvR59H-nP9hKMX7GCiV-VLad_GWDIW_FcCCFMpkcgcTezSgV96wUwV8OiyBu_MXLyx7XYH0OEIdzxIXNm9ZKjGvzU39jrgMoqHo7KAC3NXOZEw
ca.crt: 1099 bytes
namespace: 11 bytes
[root@k8s-master01 ~/k8s-ha-install/dashboard]#
配置修改
将Kube-proxy改为ipvs模式，因为在初始化集群的时候注释了ipvs配置，所以需要自行修改一下： 在master01节点执行
[root@k8s-master01 ~/k8s-ha-install/dashboard]# kubectl edit cm kube-proxy -n kube-system
mode： 修改为 mode: ipvs
更新Kube-Proxy的Pod：
[root@k8s-master01 ~/k8s-ha-install/dashboard]# kubectl patch daemonset kube-proxy -p "\\"spec\\":\\"template\\":\\"metadata\\":\\"annotations\\":\\"date\\":\\"`date +\'%s\'`\\"" -n kube-system
验证Kube-Proxy模式：
[root@k8s-master01 ~/k8s-ha-install/dashboard]# curl 127.0.0.1:10249/proxyMode
ipvs[root@k8s-master01 ~/k8s-ha-install/dashboard]#
注意事项：
注意：kubeadm安装的集群，证书有效期默认是一年。master节点的kube-apiserver、kube-scheduler、kube-controller-manager、etcd都是以容器运行的。可以通过kubectl get po -n kube-system查看。
启动和二进制不同的是，
kubelet的配置文件在/etc/sysconfig/kubelet和/var/lib/kubelet/config.yaml
其他组件的配置文件在/etc/Kubernetes/manifests目录下，比如kube-apiserver.yaml，该yaml文件更改后，kubelet会自动刷新配置，也就是会重启pod。不能再次创建该文件
Kubeadm安装后，因为污点的原因，master节点默认不允许部署pod，可以通过以下方式打开：
#查看Taints：
[root@k8s-master01 ~]# kubectl describe node -l node-role.kubernetes.io/master= | grep Taints
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
Taints: node-role.kubernetes.io/master:NoSchedule
删除Taint：
[root@k8s-master01 ~]# kubectl taint node -l node-role.kubernetes.io/master node-role.kubernetes.io/master:NoSchedule-
node/k8s-master01 untainted
node/k8s-master02 untainted
node/k8s-master03 untainted
[root@k8s-master01 ~]# kubectl describe node -l node-role.kubernetes.io/master= | grep Taints
Taints: <none>
Taints: <none>
Taints: <none>
以上是关于k8s/kubeadm 生产环境高可用集群部署的主要内容，如果未能解决你的问题，请参考以下文章
K8S------Kubeadm部署Kubernetes集群+Dashboard+Harbor仓库
K8S------Kubeadm部署Kubernetes集群+Dashboard+Harbor仓库