k8s集群中节点退出重入

Posted 2021-12-02 知识摆渡人

本文章基于kubeadm方式部署的k8s集群进行讲解，其他方式部署的k8s集群会有出入，不要照搬。

一、节点退出（剔除）

在硬件升级，维护或节点出现异常等情况下，我们需要将某些剔除，使其脱离kubernetes集群的调度范围。 先将要剔除会退出的节点设置成不可调度状态，设置好之后节点状态上会多了一个 “SchedulingDisabled”标记，这样新创建的pod就不会网该节点调度

kubectl cordon test-k8s-node05

之后驱逐调节点上的pod

kubectl drain test-k8s-node05

如果是节点出了问题，执行不了指令，可以采取强制驱逐的方式

kubectl delete pods -n kube-system nginx-6qz6s

将节点从集群中剔除或退出

kubectl delete node test-k8s-node05

二、 节点重新加入集群

首先查看有无可用的token（token24小时过期）

kubeadm token list
TOKEN     TTL       EXPIRES   USAGES    DESCRIPTION   EXTRA GROUPS

没有的话就生成一个新的token

kubeadm token create 
t7scf3.kdc0ojlfipuh56vn

获取CA证书 sha256 编码 hash 值

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed s/^.* //
99b9ae18c2cec0e9f55ae90c15f0c27e6748994fc6873affd7b07ac1d8362101

由于该节点原来已经加入过集群，只是后来从集群中退出了，所以需要清理下原来的信息，否则会报错：

[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
        [ERROR FileAvailable--etc-kubernetes-bootstrap-kubelet.conf]: /etc/kubernetes/bootstrap-kubelet.conf already exists
        [ERROR Port-10250]: Port 10250 is in use
        [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

执行如下命令清理旧的信息（如果是新节点这步省略）

kubeadm reset

在该节点上用上面生成的token和CA证书 sha256 编码 hash 值，将node节点加入集群

kubeadm join 192.168.2.123:6443 --token t7scf3.kdc0ojlfipuh56vn --discovery-token-ca-cert-hash sha256:99b9ae18c2cec0e9f55ae90c15f0c27e6748994fc6873affd7b07ac1d8362101


[preflight] Running pre-flight checks
        [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with kubectl -n kube-system get cm kubeadm-config -oyaml
[kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...


This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.


Run kubectl get nodes on the control-plane to see this node join the cluster.

查看节点是否加入

kubectl top nodes
NAME              CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
........
test-k8s-node01   2283m        7%     26679Mi         83%       
test-k8s-node02   1949m        12%    22489Mi         70%       
test-k8s-node03   1851m        11%    18702Mi         58%       
test-k8s-node04   1102m        27%    9699Mi          61%       
test-k8s-node05   496m         12%    6858Mi          43% 
........