k8s1.26.x 使用kubeadm 的cri-dockerd 部署

Posted 2023-01-05

标签（空格分隔）： kubernetes系列

---

一：k8s1.26.x 的概述：

Kubernetes v1.26 是 2022 年的最后一个大版本更新，包含了 37 项主要的更新。

8 月底正式发布的 v1.25 中包含 40 项，单从这个角度来看，Kubernetes 每个版本节奏的把控也是挺不错的。

在 v1.26 中包含了很多直接影响用户与 Kubernetes 交互相关的功能，比如以下内容中会介绍的允许跨 namespace 持久化卷快照引用；允许用户无需额外的 exporter 即可自定义 Service Level Indicator(SLI); 

kubectl events 达到 Beta；支持 Pod 所需资源就绪后再进行调度；支持使用 OpenAPI v3 等。

以及针对高性能负载运行在哪个物理 CPU 的场景支持。

更多介绍：
 https://github.com/kubernetes-sigs/vsphere-csi-driver

 https://github.com/kubernetes-sigs/azurefile-csi-driver

 https://github.com/kubernetes/enhancements/issues/1489

 https://k8s.io/releases/download/

 https://kubernetes.io/docs/tutorials/

二：k8s 1.26.x 部署

2.1 系统环境准备

系统：
    CentOS7.9x64
主机名：
   cat /etc/hosts 
-----
172.16.10.11    flyfish11
172.16.10.12    flyfish12
172.16.10.13    flyfish13
172.16.10.14    flyfish14
172.16.10.15    flyfish15
172.16.10.16    flyfish16
172.16.10.17    flyfish17

------

本次部署前3台 flyfish11做master 其它2台 做slaved 的 node 节点

系统关闭firwalld，selinux ,清空防火墙iptables 

2.2 系统初始化

1. 所有节点上关闭swap分区
swapoff -a ; sed -i /fstab/d /etc/fstab 

2. 升级系统内核

导入elrepo gpg key
 rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

安装elrepo YUM源仓库
 yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

安装kernel-ml版本，ml为长期稳定版本，lt为长期维护版本
 yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64

设置grub2默认引导为0
 grub2-set-default 0

重新生成grub2引导文件
 grub2-mkconfig -o /boot/grub2/grub.cfg

更新后，需要重启，使用升级的内核生效。
 reboot

重启后，需要验证内核是否为更新对应的版本
 uname -r

3.添加网桥过滤及内核转发配置文件

cat <<EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness=0
EOF

#加载br_netfilter模块
# modprobe br_netfilter

#使之生效
sysctl -p /etc/sysctl.d/k8s.conf 

#查看是否加载
# lsmod | grep br_netfilter
br_netfilter           22256  0

4. 安装ipset及ipvsadm

安装ipset及ipvsadm
yum -y install ipset ipvsadm

#配置ipvsadm模块加载方式.添加需要加载的模块

cat > /etc/sysconfig/modules/ipvs.module <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_sh
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- nf_conntrack
EOF

授权、运行、检查是否加载
chmod 755 /etc/sysconfig/modules/ipvs.module &&  /etc/sysconfig/modules/ipvs.module

lsmod | grep -e ip_vs -e nf_conntrack

2.3 Docker安装准备

Docker安装环境准备
## 安装一些必要工具

yum install -y yum-utils device-mapper-persistent-data lvm2

## 配置docker镜像源

wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo


## 查看所有的可用版本
yum list docker-ce --showduplicates | sort -r

#安装旧版本 yum install docker-ce-cli-19.03.15-3.el7  docker-ce-19.03.15-3.el7

安装源里最新版本
 yum install docker-ce

## 镜像加速器
mkdir -p /etc/docker

cat > /etc/docker/daemon.json <<EOF

   "registry-mirrors": ["https://uwtwp6l0.mirror.aliyuncs.com"],
    "exec-opts": ["native.cgroupdriver=systemd"]

EOF

#启动Docker服务

systemctl enable --now docker
systemctl start docker

2.4 部署cri-dockerd

到下面的链接下载最新版cri-docker
https://github.com/Mirantis/cri-dockerd/tags

下载：cri-dockerd-0.3.0.amd64.tgz

所有节点 都安装 cri-dockerd

# 拷贝二进制文件

# tar -xf cri-dockerd-0.3.0.amd64.tgz 
# cp cri-dockerd/cri-dockerd /usr/bin/
# chmod +x /usr/bin/cri-dockerd 


# 配置启动文件

cat <<"EOF" > /usr/lib/systemd/system/cri-docker.service
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket

[Service]
Type=notify

ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7

ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always

StartLimitBurst=3

StartLimitInterval=60s

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

TasksMax=infinity
Delegate=yes
KillMode=process

[Install]
WantedBy=multi-user.target

EOF

# 生成socket 文件

cat <<"EOF" > /usr/lib/systemd/system/cri-docker.socket
[Unit]
Description=CRI Docker Socket for the API
PartOf=cri-docker.service

[Socket]
ListenStream=%t/cri-dockerd.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

EOF

# 启动cri-docker并设置开机自动启动

systemctl daemon-reload ; systemctl enable cri-docker --now

systemctl is-active cri-docker

----
active
----

2.5 安装kubernetes

阿里云YUM源【国内主机】
 cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum clean all && yum makecache


所有节点均可安装

# 查看所有的可用版本
$  yum list  kubeadm  kubelet kubectl --showduplicates | sort -r



# 默认安装的版本就是最新版1.26.X，当然也可以指定版本安装 ，如 yum install kubelet-1.16.2 kubeadm-1.16.2 kubectl-1.16.2
$ yum install  kubeadm  kubelet kubectl

#安装后查看版本
$ kubeadm version

设置kubelet为开机自启动即可，由于没有生成配置文件，集群初始化后自动启动
$ systemctl enable kubelet




为了实现docker使用的cgroupdriver与kubelet使用的cgroup的一致性，建议修改如下文件内容。
cat <<EOF > /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--cgroup-driver=systemd"
EOF

$ systemctl start kubelet

$ systemctl is-active kubelet
----
activeing
-----

2.6 初始化集群

初始化集群

kubeadm init --image-repository registry.aliyuncs.com/google_containers --kubernetes-version=v1.26.0 --pod-network-cidr=10.244.0.0/16 --cri-socket /var/run/cri-dockerd.sock

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.16.10.11:6443 --token oqeo4p.5ma3omzmep3o7ske \\
        --discovery-token-ca-cert-hash sha256:665bdac90d2f8adb1abfe19a2b9eff81b356c07c03f6600d2690587e9d51d7b1

2.7 其它worker 节点加入集群

其它worker节点加入
kubeadm join 172.16.10.11:6443 --token oqeo4p.5ma3omzmep3o7ske \\
        --discovery-token-ca-cert-hash sha256:665bdac90d2f8adb1abfe19a2b9eff81b356c07c03f6600d2690587e9d51d7b1 --cri-socket /var/run/cri-dockerd.sock 

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl get node 

2.8 配置 calico 网络插件

下载 calico 插件：
wget https://docs.projectcalico.org/manifests/calico.yaml --no-check-certificate

修改 网段
vim calico.yaml
-----

 # no effect. This should fall within `--cluster-cidr`.
 - name: CALICO_IPV4POOL_CIDR
   value: "10.244.0.0/16"
 # Disable file logging so `kubectl logs` works.

------

下载镜像：

cat calico.yaml |grep image

image: docker.io/calico/cni:v3.24.5
image: docker.io/calico/node:v3.24.5
image: docker.io/calico/kube-controllers:v3.24.5

-------------------------

docker pull docker.io/calico/cni:v3.24.5

docker pull docker.io/calico/node:v3.24.5

docker pull docker.io/calico/kube-controllers:v3.24.5

kubectl apply -f calico.yaml

kubectl get pod -n kube-system

kubect get node 

2.9 去掉master 节点的污点

#查看污点 
kubectl describe node flyfishsrvs01 | grep -i taint
Taints:             node-role.kubernetes.io/control-plane:NoSchedule

#去除污点
kubectl taint node flyfish11 node-role.kubernetes.io/control-plane:NoSchedule- 
 

2.10 部署dashborad

wget https://raw.githubusercontent.com/cby-chen/Kubernetes/main/yaml/dashboard.yaml
目前最新版本v2.7.0 

vim dashboard.yaml

----
spec:
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 30001
  type: NodePort
  selector:
    k8s-app: kubernetes-dashboard
----


kubectl apply -f dashborad.yaml

kubectl get pods -n kubernetes-dashboard
kubectl get pods,svc -n kubernetes-dashboard

创建用户：
wget https://raw.githubusercontent.com/cby-chen/Kubernetes/main/yaml/dashboard-user.yaml

kubectl apply -f dashboard-user.yaml

创建token 
kubectl -n kubernetes-dashboard create token admin-user

3.7 登录浏览器访问

https://172.16.10.11:30001
输入token：
----
eyJhbGciOiJSUzI1NiIsImtpZCI6IkF2Rlg4Q1ZZemV0aVRyTklDc0JrRHVYV3c4eGFXRzVpYml1bm1yWlIzbzAifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiXSwiZXhwIjoxNjYzOTE4ODcxLCJpYXQiOjE2NjM5MTUyNzEsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5jbHVzdGVyLmxvY2FsIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJrdWJlcm5ldGVzLWRhc2hib2FyZCIsInNlcnZpY2VhY2NvdW50Ijp7Im5hbWUiOiJhZG1pbi11c2VyIiwidWlkIjoiNDNlY2FlN2MtMmZjMi00ZGRkLTgzNzQtZjQyZTczMmQyZDhjIn19LCJuYmYiOjE2NjM5MTUyNzEsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlcm5ldGVzLWRhc2hib2FyZDphZG1pbi11c2VyIn0.Kw0Ys19fyfkBoH_BOY2QpaMcklddJll9odZXB_RaEAHBjDS3S_GeJX4kA4WnVUM80P5OlTvjjI13GS2vvSeLuROHknZINJEDUdSZDS5DuBrRlvvw_cR3gD5D3Wb_4rvEzCrgzDrbZDz-vaLv5m1zcTSUKBgdh2vqB2gNt1jAx8K3ujS4tI7A0HohsoHI89YN3tINeTXPG8x0V9-u7qTFrf6O-f--sLsAK5NJhnwvqkTXgCqdNwlCuWy_K0ga3-hOZdJeCvm0r_gAEpyogk_9qZjagQmyIAGOI79RWz_JdH7vPTLC5AsBQ-SSbyDo_mFos1r1aRLbgwhkJE5DdPiYfA
----