云原生kubeadm部署单master集群（contained运行时）

Posted 2022-08-26 键客李大白

一、部署说明

主机清单

版本说明

• contained version：1.6.5

• kubeadm version: 1.23.9

• kubectl version: 1.23.9
• kubelet version: 1.23.9

二、主机初始化

配置/etc/hosts

$ cat <<EOF >> /etc/hosts
192.168.2.60 kubeadm-master1
192.168.2.61 kubeadm-node01
EOF

配置yum仓库

配置清华镜像站，bash-completion：命令table键

$ cat <<EOF >/etc/yum.repos.d/kubernetes.repo 
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.tuna.tsinghua.edu.cn/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
EOF 
$ yum clean all && yum makecache  && yum repolist
$ yum install -y vim net-tools  bash-completion wget lrzsz     #下载常用的软件

配置时间同步（ntpdate)

$ yum install -y ntpdate
$ ntpdate  ntp.aliyun.com        #同步的时间服务器地址
$ systemctl restart ntpdate.service  && systemctl status ntpdate.service

内核升级

必须操作，内核太低会出现很多问题，需要上传kernel内核包进行安装升级

$ grub2-install  /dev/sda
$ rpm -ivh  kernel-ml-4.19.12-1.el7.elrepo.x86_64.rpm
$ vim /etc/default/grub               #修改内核启动顺序
 GRUB_DEFAULT=0                      
#GRUB_DEFAULT=saved中，将saved修改为0
$ sed -i /GRUB_DEFAULT/s/saved/0/ /etc/default/grub  &&  grep GRUB_DEFAULT   /etc/default/grub

重新编译内核启动文件

$  grub2-mkconfig -o /boot/grub2/grub.cfg

查看默认启动的内核

$ awk -F\\ $1=="menuentry " print i++ " : " $2 /etc/grub2.cfg 
0 : CentOS Linux (4.19.12-1.el7.elrepo.x86_64) 7 (Core)
1 : CentOS Linux (3.10.0-862.el7.x86_64) 7 (Core)
2 : CentOS Linux (0-rescue-35ac0fa1f7924eb18b1c0697c294d34d) 7 (Core)

重启主机后查看内核版本

$ reboot 
$ uname -r
4.19.12-1.el7.elrepo.x86_64

注：内核升级完成后需要重启主机，然后使用uname -r查看内核版本是否升级成功。

配置内核参数

在内核版本升级后操作。

$ cat >> /etc/sysctl.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF    
$ sysctl -p

• net.bridge.bridge-nf-call-iptables：开启桥设备内核监控（ipv4）

• net.ipv4.ip_forward：开启路由转发

• net.bridge.bridge-nf-call-ip6tables：开启桥设备内核监控（ipv6）

以上3项为必须参数，其他参数可根据需要添加。

配置IPVS

$ yum install  -y ipvsadm ipset sysstat conntrack libseccomp  
$ cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack ip_tables ip_set xt_set ipt_set ipt_rpfilter ipt_REJECT ipip "
for kernel_module in \\$ipvs_modules; do
  /sbin/modinfo -F filename \\$kernel_module > /dev/null 2>&1
  if [ $? -eq 0 ]; then
    /sbin/modprobe \\$kernel_module
  fi
done
EOF  
$ chmod 755 /etc/sysconfig/modules/ipvs.modules 
$ sh /etc/sysconfig/modules/ipvs.modules 
$ lsmod | grep ip_vs

dummy0网卡和kube-ipvs0网卡：在安装k8s集群时，启用了ipvs的话，就会有这两个网卡。（将service的IP绑定在kube-ipvs0网卡上）

下载并安装kubeadm、kubectl、kubelet

下载rpm安装包

$ mkdir kubeadm-install    #创建存放rpm安装包的目录
$ yum install   kubeadm-1.23.9 kubectl-1.23.9 kubelet-1.23.9 --downloadonly --downloaddir=./kubeadm-install

安装

$ yum  install -y kubeadm-install/*.rpm
$ rpm -qa |  grep kube
kubernetes-cni-0.8.7-0.x86_64
kubectl-1.23.9-0.x86_64
kubelet-1.23.9-0.x86_64
kubeadm-1.23.9-0.x86_64

启动kubelet

$ systemctl enable --now kubelet && systemctl status kubelet

设置Table键补全

让命令可用自动table键进行补全，对新手无法记住命令提供很好的支持，所在主机进行该操作方可使用table补全。

• Kubectl命令补全：

$ kubectl completion  bash >  /etc/bash_completion.d/kubelet

• Kubeadm命令补全：

$ kubeadm  completion  bash >  /etc/bash_completion.d/kubeadm

安装Containerd(二进制)

1）下载二进制包

$ wget https://github.com/containerd/containerd/releases/download/v1.6.5/cri-containerd-1.6.5-linux-amd64.tar.gz
$ tar zxvf cri-containerd-1.6.5-linux-amd64.tar.gz
$ ls  -l
drwxr-xr-x  4 root root        51 4月  26 07:52 etc
drwxr-xr-x  4 root root        35 4月  26 07:51 opt
drwxr-xr-x  3 root root        19 4月  26 07:50 usr

etc目录：主要为containerd服务管理配置文件及cni虚拟网卡配置文件；

opt目录：主要为gce环境中使用containerd配置文件及cni插件；

usr目录：主要为containerd运行时的二进制文件，包含runc；

2）拷贝二进制可执行文件到$PATH中

$ ls usr/local/bin/
containerd  containerd-shim  containerd-shim-runc-v1  containerd-shim-runc-v2  containerd-stress  crictl  critest  ctd-decoder  ctr
$ cp usr/local/bin/*  /usr/local/bin/

3）创建初始配置文件

Containerd 的默认配置文件为 /etc/containerd/config.toml

$ mkdir -p /etc/containerd/
$ containerd config default > /etc/containerd/config.toml    #创建默认的配置文件

4）修改配置

• 替换镜像源

由于国内环境原因我们需要将 sandbox_image 镜像源设置为阿里云google_containers镜像源。

$ sed -i "s#k8s.gcr.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g"  /etc/containerd/config.toml
#等同于：
$ vim /etc/containerd/config.toml
  [plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

• 配置镜像加速

$ vim /etc/containerd/config.toml
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["https://xlx9erfu.mirror.aliyuncs.com"]

• 配置驱动器

​ Containerd 和 Kubernetes 默认使用旧版驱动程序来管理 cgroups，但建议在基于 systemd 的主机上使用该驱动程序，以符合 cgroup 的“单编写器”规则。

$ sed -i s#SystemdCgroup = false#SystemdCgroup = true#g /etc/containerd/config.toml
#等同于
$ vim /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
  SystemdCgroup = true

5）创建服务管理配置文件

​ 拷贝服务管理配置文件到/usr/lib/systemd/system/目录

$ grep -v ^# etc/systemd/system/containerd.service
$ mv etc/systemd/system/containerd.service /usr/lib/systemd/system/containerd.service

6）启动 containerd 服务

$ systemctl daemon-reload
$ systemctl enable --now containerd.service
$ systemctl status containerd.service
$ containerd  --version    #查看版本
containerd github.com/containerd/containerd v1.6.3 f830866066ed06e71bad64871bccfd34daf6309c

7）安装runc（二进制）

由于二进制包中提供的runC默认需要系统中安装seccomp支持，需要单独安装，且不同版本runC对seccomp版本要求不一致，所以建议单独下载runC 二进制包进行安装，里面包含了seccomp模块支持。

Runc是真正运行容器的工具

$ mv usr/local/sbin/runc  /usr/bin/
$ runc -version
runc: symbol lookup error: runc: undefined symbol: seccomp_notify_respond

【报错原因】缺少依赖包libseccomp（2.4以上版本）安全计算模式。解决办法：

$ wget http://rpmfind.net/linux/centos/8-stream/BaseOS/x86_64/os/Packages/libseccomp-2.5.1-1.el8.x86_64.rpm
$ rpm -ivh libseccomp-2.5.1-1.el8.x86_64.rpm
$ rpm -qa |  grep libseccomp
libseccomp-2.5.1-1.el8.x86_64
$ runc  -version
runc version 1.1.2
commit: v1.1.2-0-ga916309f
spec: 1.0.2-dev
go: go1.17.11
libseccomp: 2.5.1

配置crictl客户端

$ mv etc/crictl.yaml  /etc/
$ cat  /etc/crictl.yaml 
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: false

三、初始化master节点

3.1 检测安装环境

检测主机环境是否达到集群的要求，可根据结果提示进行逐一排除故障

$ kubeadm init --dry-run

3.2 镜像下载/导入

1）列出需要使用的镜像列表

$ kubeadm config images list

2）拉取镜像

$ kubeadm config images pull  \\
--image-repository  registry.aliyuncs.com/google_containers \\
--kubernetes-version  v1.23.10

--image-repository: 从哪个地方下载镜像（默认"k8s.gcr.io"，但k8s.gcr.io国内无法访问）；

--kubernetes-version: 指定kubernetes集群的镜像版本；

3）修改镜像tag

$ crictl  images
IMAGE                         TAG                IMAGE ID          SIZE
registry.aliyuncs.com/google_containers/coredns   v1.8.6        a4ca41631cc7a       13.6MB
registry.aliyuncs.com/google_containers/etcd     3.5.1-0        25f8c7f3da61c      98.9MB
registry.aliyuncs.com/google_containers/kube-apiserver    v1.23.10   9ca5fafbe8dc1       32.6MB
registry.aliyuncs.com/google_containers/kube-controller-manager   v1.23.10    91a4a0d5de4e9   30.2MB
registry.aliyuncs.com/google_containers/kube-proxy   v1.23.10      71b9bf9750e1f       39.3MB
registry.aliyuncs.com/google_containers/kube-scheduler     v1.23.10      d5c0efb802d95   15.1MB
registry.aliyuncs.com/google_containers/pause    3.6      6270bb605e12e       302kB

如果报错：

$ crictl  images
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. 
ERRO[0000] unable to determine image API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" 

解决：配置/etc/crictl.yaml。

3.3 创建配置文件

创建默认的配置文件

$ kubeadm   config  print  init-defaults > kubeadm-config.yaml  

修改配置文件

$ cat kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 192.168.2.60  #修改为控制节点IP（VIP）
  bindPort: 6443
nodeRegistration:
  criSocket: /run/containerd/containerd.sock  #使用containerd为容器运行时
  imagePullPolicy: IfNotPresent
  name: kubeadm-master1     #修改为控制节点主机名
  taints: null
---
apiServer:
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: 
dns: 
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers  #修改为阿里镜像地址
kind: ClusterConfiguration
kubernetesVersion: 1.23.9  #版本
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16   #指定Pod网段
  serviceSubnet: 10.96.0.0/12  #指定Service网段
scheduler: 
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind:  KubeProxyConfiguration
mode: ipvs
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd

3.4 初始化

$ kubeadm init --config kubeadm-config.yaml
[init] Using Kubernetes version: v1.23.9
[preflight] Running pre-flight checks
    [WARNING Swap]: swap is enabled; production deployments should disable swap unless testing the NodeSwap feature gate of the kubelet
[preflight] Pulling images required for setting up a Kubernetes cluster
...
...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.2.60:6443 --token abcdef.0123456789abcdef \\
    --discovery-token-ca-cert-hash sha256:504b652bca396c4dccfee3dac285e3127f9f8b5033798786f39fbf78b6301515 

3.5 环境配置

根据初始化成功后的提示对集群进行基础的配置。

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ export KUBECONFIG=/etc/kubernetes/admin.conf
$ echo "KUBECONFIG=/etc/kubernetes/admin.conf" >>  ~/.bashrc && source  ~/.bashrc
$ kubectl get nodes
NAME              STATUS     ROLES                  AGE    VERSION
kubeadm-master1   NotReady   control-plane,master   3m3s   v1.23.9

3.6 集群重置（可选）

初始化master有问题，可将集群重置，再次初始化master

$ kubeadm reset

四、node节点加入集群

node节点完成主机初始化后，执行以下命令即可将节点加入集群：

kubeadm join 192.168.2.60:6443 --token abcdef.0123456789abcdef \\
    --discovery-token-ca-cert-hash sha256:504b652bca396c4dccfee3dac285e3127f9f8b5033798786f39fbf78b6301515 

五、安装插件

5.1 安装Calico网络插件

$ wget https://docs.projectcalico.org/v3.18/manifests/calico.yaml --no-check-certificate
$ kubectl apply -f calico.yaml

5.2 安装CoreDNS域名解析插件

​ 在kubeadm安装的k8s集群中，安装Calico网络插件后会自动安装CoreDNS插件。

5.3 安装Metrics数据采集插件

$ vim /etc/kubernetes/manifests/kube-apiserver.yaml
- --enable-aggregator-routing=true
$ wget  https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.5.2/components.yaml
$ kubectl apply   -f components.yaml
$ grep image components.yaml
 image: k8s.gcr.io/metrics-server/metrics-server:v0.5.2
$ crictl pull registry.aliyuncs.com/google_containers/metrics-server:v0.5.2