离线部署k8s-1.19.0集群

Posted 2021-06-23 无荨

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了离线部署k8s-1.19.0集群相关的知识，希望对你有一定的参考价值。

0、节点说明：

1、环境配置

1.1 关闭防火墙、selinux、swap

setenforce 0
sed -i \'s/=enforcing/=disabled/g\' /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld

1.2 做免密操作

生成密钥（master）：
sed -i \'35cStrictHostKeyChecking no\'  /etc/ssh/ssh_config
ssh-keygen -t rsa -f /root/.ssh/id_rsa -P ""
cp /root/.ssh/id_rsa.pub  /root/.ssh/authorized_keys
发送到其他节点：
[root@k8s-master ~]# scp -r /root/.ssh/ root@192.168.160.130:/root
[root@k8s-master ~]# scp -r /root/.ssh/ root@192.168.160.131:/root

免密测试：

免密测试：
[root@k8s-master ~]# ssh 192.168.160.131
Last login: Fri May 21 10:04:36 2021 from 192.168.160.129
[root@k8s-node2 ~]# ssh 192.168.160.130
Last login: Fri May 21 09:12:10 2021 from 192.168.160.1
[root@k8s-node1 ~]# ssh 192.168.160.129
The authenticity of host \'192.168.160.129 (192.168.160.129)\' can\'t be established.
ECDSA key fingerprint is SHA256:FSe5JBJyY0olAkh+sfW3uOj1fQ+6eCXR4F5meZLvrp4.
ECDSA key fingerprint is MD5:50:44:e3:e2:35:5d:7f:68:9e:7e:63:b7:d4:e6:dd:6c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added \'192.168.160.129\' (ECDSA) to the list of known hosts.
Last login: Fri May 21 09:12:31 2021 from 192.168.160.1
[root@k8s-master ~]#

免密测试

1.3 设置主机名解析

#设置主机名：hostnamectl set-hostname HOSTNAME
#配置主机名解析：
cat > /etc/hosts << QQQ
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.160.129  k8s-master
192.168.160.130  k8s-node1
192.168.160.131  k8s-node2
QQQ
#发送到其他节点
[root@k8s-master ~]# scp /etc/hosts 192.168.160.130:/etc
[root@k8s-master ~]# scp /etc/hosts 192.168.160.131:/etc

1.4 关闭swap交换分区

[root@k8s-master ~]# swapoff -a && sysctl -w vm.swappiness=0
vm.swappiness = 0
[root@k8s-master ~]# ssh 192.168.160.130 "swapoff -a && sysctl -w vm.swappiness=0"
vm.swappiness = 0
[root@k8s-master ~]# ssh 192.168.160.131 "swapoff -a && sysctl -w vm.swappiness=0"
vm.swappiness = 0
[root@k8s-master ~]# sed -ri \'s/.*swap.*/#&/\' /etc/fstab
[root@k8s-master ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Thu May 20 17:07:46 2021
#
# Accessible filesystems, by reference, are maintained under \'/dev/disk\'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=e22ac659-213c-476a-9ecb-6e6b9d5e7fba /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0
/dev/cdrom /iso iso9660 defaults 0 0
/dev/cdrom /iso iso9660 defaults 0 0
[root@k8s-master ~]#

验证swap分区关闭情况：

swap验证

1.5 (选项)配置yum源：（如果公司有自己的yum，使用自己的yum源）

[root@k8s-master ~]# mkdir /etc/yum.repos.d/bak
[root@k8s-master ~]# mount /dev/cdrom /iso
[root@k8s-master ~]# echo "/dev/cdrom /iso iso9660 defaults 0 0">>/etc/fstab
[root@k8s-master ~]# mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak
[root@k8s-master ~]# cat>/etc/yum.repos.d/iso.repo <<QQQ
> [iso]
> name=iso
> baseurl=file:///iso
> enabled=1
> gpgcheck=0
> QQQ
[root@k8s-master ~]# mkdir /iso
[root@k8s-master ~]# mount /dev/cdrom /iso
mount: /dev/sr0 写保护，将以只读方式挂载
[root@k8s-master ~]# echo "/dev/cdrom /iso iso9660 defaults 0 0">>/etc/fstab
[root@k8s-master ~]# yum -y install vim net-tools unzip

配置本地yum

1.6 安装docker （内核版本不同，安装的rpm包不同）

[root@k8s-master k8s]# scp -r /root/k8s 192.168.160.130:/root
[root@k8s-master k8s]# scp -r /root/k8s 192.168.160.131:/root
[root@k8s-master ~]# ls /root/k8s/docker/docker-rpm/
containerd.io-1.4.4-3.1.el7.x86_64.rpm                docker-scan-plugin-0.7.0-3.el7.x86_64.rpm
container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm  fuse3-libs-3.6.1-4.el7.x86_64.rpm
docker-ce-20.10.6-3.el7.x86_64.rpm                    fuse-overlayfs-0.7.2-6.el7_8.x86_64.rpm
docker-ce-cli-20.10.6-3.el7.x86_64.rpm                slirp4netns-0.4.3-4.el7_8.x86_64.rpm
docker-ce-rootless-extras-20.10.6-3.el7.x86_64.rpm
[root@k8s-master ~]# cd /root/k8s/docker/docker-rpm
[root@k8s-master docker-rpm]# yum -y localinstall ./*
[root@k8s-master docker-rpm]# cd ..
[root@k8s-master2 docker]# ls
docker-rpm  docker-speed.sh
[root@k8s-master docker]# sh docker-speed.sh
{
        "exec-opts": ["native.cgroupdriver=systemd"]
}
Created symlink from /etc/systemd/system/multi-user.target.wants/docker.service to /usr/lib/systemd/system/docker.service.

#无外网可以忽略/etc/docker/daemon.json
[root@k8s-master docker]# cat docker-speed.sh
#!/bin/bash
sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-\'EOF\'
{
        "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker
systemctl enable docker

1.7 要保证打开内置的桥功能，这个是借助于iptables来实现的

[root@k8s-master ~]# echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables
[root@k8s-master ~]# echo 1 >/proc/sys/net/ipv4/ip_forward

2.开始部署master节点

2.1 安装kubectl、kubeadm、kubelet，并且设置kubelet开机自启

[root@k8s-master k8s-rpm]# ls
conntrack-tools-1.4.4-7.el7.x86_64.rpm  kubectl-1.19.0-0.x86_64.rpm        libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm
cri-tools-1.13.0-0.x86_64.rpm           kubelet-1.19.0-0.x86_64.rpm        libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm
kubeadm-1.19.0-0.x86_64.rpm             kubernetes-cni-0.8.7-0.x86_64.rpm  libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm
[root@k8s-master k8s-rpm]# yum localinstall -y ./*
…….
[root@k8s-master ~]# systemctl enable kubelet
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /usr/lib/systemd/system/kubelet.service.

kubelet 运行在 Cluster 所有节点上，负责启动 Pod 和容器。
kubeadm 用于初始化 Cluster。
kubectl 是 Kubernetes 命令行工具。通过 kubectl 可以部署和管理应用，查看各种资源，创建、删除和更新各种组件。

2.2 初始化集群

2.2.1 拉取镜像：

[root@k8s-master k8s-images]# ll
总用量 1003112
-rw-r--r--. 1 root root  45365760 5月  20 13:45 coredns-1.7.0.tar.gz
-rw-r--r--. 1 root root 225547264 5月  20 13:45 dashboard-v2.0.1.tar.gz
-rw-r--r--. 1 root root 254629888 5月  20 13:45 etcd-3.4.9-1.tar.gz
-rw-r--r--. 1 root root  65271296 5月  20 13:45 flannel-v0.13.1-rc2.tar.gz
-rw-r--r--. 1 root root 120040960 5月  20 13:45 kube-apiserver-v1.19.0.tar.gz
-rw-r--r--. 1 root root 112045568 5月  20 13:45 kube-controller-manager-v1.19.0.tar.gz
-rw-r--r--. 1 root root 119695360 5月  20 13:45 kube-proxy-v1.19.0.tar.gz
-rw-r--r--. 1 root root  46919168 5月  20 13:45 kube-scheduler-v1.19.0.tar.gz
-rw-r--r--. 1 root root    692736 5月  20 13:44 pause-3.2.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/kube-apiserver-v1.19.0.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/coredns-1.7.0.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/dashboard-v2.0.1.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/etcd-3.4.9-1.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/flannel-v0.13.1-rc2.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/kube-controller-manager-v1.19.0.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/kube-proxy-v1.19.0.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/kube-scheduler-v1.19.0.tar.gz
[root@k8s-master ~]# docker load -i /root/k8s/k8s-images/pause-3.2.tar.gz

拉取本地镜像

2.2.2 修改镜像名称并删除旧镜像

#使用kubeadm config images list查看需要tag的镜像版本

[root@k8s-master ~]# kubeadm config images list
I0521 13:38:41.768238   12611 version.go:252] remote version is much newer: v1.21.1; falling back to: stable-1.19
W0521 13:38:45.604893   12611 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
k8s.gcr.io/kube-apiserver:v1.19.11
k8s.gcr.io/kube-controller-manager:v1.19.11
k8s.gcr.io/kube-scheduler:v1.19.11
k8s.gcr.io/kube-proxy:v1.19.11
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.9-1
k8s.gcr.io/coredns:1.7.0
#通过kubeadm config images list修改镜像名称：
[root@k8s-master ~]# docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.19.0 k8s.gcr.io/kube-apiserver:v1.19.11
[root@k8s-master ~]# docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.19.0 k8s.gcr.io/kube-controller-manager:v1.19.11
[root@k8s-master ~]# docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.19.0 k8s.gcr.io/kube-scheduler:v1.19.11
[root@k8s-master ~]# docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.19.0 k8s.gcr.io/kube-proxy:v1.19.11
[root@k8s-master ~]# docker tag registry.cn-hangzhou.aliyuncs.com/k8sos/flannel:v0.13.1-rc2 quay.io/coreos/flannel:v0.11.0-amd64
[root@k8s-master ~]# docker tag registry.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:v3.2
[root@k8s-master ~]# docker tag registry.aliyuncs.com/google_containers/etcd:3.4.9-1 k8s.gcr.io/etcd:3.4.9-1
[root@k8s-master ~]# docker tag registry.aliyuncs.com/google_containers/coredns:1.7.0 k8s.gcr.io/coredns:v1.7.0
[root@k8s-master ~]# docker tag kubernetesui/dashboard:v2.0.1  k8s.gcr.io/dashboard:v2.0.1
[root@k8s-master ~]# docker images
REPOSITORY                           TAG             IMAGE ID       CREATED         SIZE
quay.io/coreos/flannel               v0.11.0-amd64   60e169ce803f   3 months ago    64.3MB
k8s.gcr.io/kube-proxy                v1.19.11        bc9c328f379c   8 months ago    118MB
k8s.gcr.io/kube-apiserver            v1.19.11        1b74e93ece2f   8 months ago    119MB
k8s.gcr.io/kube-controller-manager   v1.19.11        09d665d529d0   8 months ago    111MB
k8s.gcr.io/kube-scheduler            v1.19.11        cbdc8369d8b1   8 months ago    45.7MB
k8s.gcr.io/etcd                      3.4.9-1         d4ca8726196c   10 months ago   253MB
k8s.gcr.io/coredns                   1.7.0           bfe3a36ebd25   11 months ago   45.2MB
k8s.gcr.io/dashboard                 v2.0.1          85d666cddd04   12 months ago   223MB
k8s.gcr.io/pause                     3.2             80d28bedfe5d   15 months ago   683kB

修改镜像名称

2.2.3 初始化集群

[root@k8s-master ~]# kubeadm init --apiserver-advertise-address 192.168.160.129 --pod-network-cidr=10.244.0.0/16
...
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.160.129:6443 --token 8w1usi.xrk1kgpghbn7vo66 \\
    --discovery-token-ca-cert-hash sha256:8d6937dc0c3174bbc7ff95d5c1b3cc487027007cc782522e63dd3d2ac7b45787
[root@k8s-master ~]#   mkdir -p $HOME/.kube
[root@k8s-master ~]#   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8s-master ~]#   sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-master ~]#

2.2.4 配置网络

[root@k8s-master k8s-conf]# kubectl apply -f kube-flannel.yml
podsecuritypolicy.policy/psp.flannel.unprivileged created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/flannel created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds-amd64 created
daemonset.apps/kube-flannel-ds-arm64 created
daemonset.apps/kube-flannel-ds-arm created
daemonset.apps/kube-flannel-ds-ppc64le created
daemonset.apps/kube-flannel-ds-s390x created
[root@k8s-master k8s-conf]#

kube-flannel.yml

2.2.5 修改controller-manager与scheduler配置文件

[root@k8s-master ~]# cd /etc/kubernetes/manifests/
[root@k8s-master manifests]# ls
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
#将/etc/kubernetes/manifests/下的kube-controller-manager.yaml和kube-scheduler.yaml设置的默认端口是0导致的，解决方式是注释掉对应的port即可
[root@k8s-master manifests]# cat kube-controller-manager.yaml|grep port
#    - --port=0
    port: 10257
    port: 1025

2.2.6 检查

[root@k8s-master manifests]# kubectl get ns
NAME              STATUS   AGE
default           Active   40m
kube-node-lease   Active   40m
kube-public       Active   40m
kube-system       Active   40m
[root@k8s-master manifests]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE             ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true"}
[root@k8s-master manifests]# kubectl get po -n kube-system
NAME                                 READY   STATUS    RESTARTS   AGE
coredns-f9fd979d6-cxhbf              1/1     Running   0          41m
coredns-f9fd979d6-vcvrb              1/1     Running   0          41m
etcd-k8s-master                      1/1     Running   0          41m
kube-apiserver-k8s-master            1/1     Running   0          41m
kube-controller-manager-k8s-master   1/1     Running   0          38m
kube-flannel-ds-amd64-cchsr          1/1     Running   0          39m
kube-proxy-xz7p5                     1/1     Running   0          41m
kube-scheduler-k8s-master            1/1     Running   0          38m
[root@k8s-master manifests]#

3、node节点加入集群

[root@k8s-node1 ~]# cd k8s/k8s-images/
[root@k8s-node1 k8s-images]# docker load -i flannel.tar.gz
[root@k8s-node1 k8s-images]# docker load -i kube-proxy.tar.gz
[root@k8s-node1 k8s-images]# docker load -i pause.tar.gz
[root@k8s-node1 k8s-images]# docker images
REPOSITORY                   TAG             IMAGE ID       CREATED         SIZE
quay.io/coreos/flannel       v0.11.0-amd64   60e169ce803f   3 months ago    64.3MB
k8s.gcr.io/kube-proxy        v1.19.11        bc9c328f379c   8 months ago    118MB
k8s.gcr.io/pause             3.2             80d28bedfe5d   15 months ago   683kB
[root@k8s-node1 test]# kubeadm join 192.168.160.129:6443 --token 8w1usi.xrk1kgpghbn7vo66 \\
>     --discovery-token-ca-cert-hash sha256:8d6937dc0c3174bbc7ff95d5c1b3cc487027007cc782522e63dd3d2ac7b45787
[preflight] Running pre-flight checks
        [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 20.10.6. Latest validated version: 19.03
        [WARNING Service-Kubelet]: kubelet service is not enabled, please run \'systemctl enable kubelet.service\'
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with \'kubectl -n kube-system get cm kubeadm-config -oyaml\'
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run \'kubectl get nodes\' on the control-plane to see this node join the cluster.

注：

master节点需要的镜像有：
k8s.gcr.io/kube-scheduler v1.19.3
k8s.gcr.io/kube-apiserver v1.19.3
k8s.gcr.io/kube-controller-manager v1.19.3
k8s.gcr.io/etcd 3.4.13-0
k8s.gcr.io/coredns 1.7.0
kubernetesui/dashboard v2.0.1
quay.io/coreos/flannel v0.13.0
k8s.gcr.io/kube-proxy v1.19.3
kubernetesui/metrics-scraper v1.0.4
k8s.gcr.io/pause 3.2

node节点需要的镜像有：
quay.io/coreos/flannel v0.13.0
k8s.gcr.io/kube-proxy v1.19.3
kubernetesui/metrics-scraper v1.0.4
k8s.gcr.io/pause 3.2

master、node所需镜像

所需镜像及其安装包：

[root@k8s-master ~]# tree k8s
k8s
├── dashboard
│   ├── dashboard-v2.0.1.tar.gz
│   ├── dashboard.yaml
│   └── metrics-scraper-v1.0.4.tar.gz
├── docker
│   ├── docker-rpm
│   │   ├── containerd.io-1.4.4-3.1.el7.x86_64.rpm
│   │   ├── container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm
│   │   ├── docker-ce-20.10.6-3.el7.x86_64.rpm
│   │   ├── docker-ce-cli-20.10.6-3.el7.x86_64.rpm
│   │   ├── docker-ce-rootless-extras-20.10.6-3.el7.x86_64.rpm
│   │   ├── docker-scan-plugin-0.7.0-3.el7.x86_64.rpm
│   │   ├── fuse3-libs-3.6.1-4.el7.x86_64.rpm
│   │   ├── fuse-overlayfs-0.7.2-6.el7_8.x86_64.rpm
│   │   └── slirp4netns-0.4.3-4.el7_8.x86_64.rpm
│   └── docker-speed.sh
├── k8s-conf
│   └── kube-flannel.yml
├── k8s-images
│   ├── coredns-1.7.0.tar.gz
│   ├── etcd-3.4.9-1.tar.gz
│   ├── flannel-v0.11.0-amd64.tar.gz
│   ├── kube-apiserver-v1.19.11.tar.gz
│   ├── kube-controller-manager-v1.19.11.tar.gz
│   ├── kube-proxy-v1.19.11.tar.gz
│   ├── kube-scheduler-v1.19.11.tar.gz
│   └── pause-3.2.tar.gz
└── k8s-rpm
    ├── conntrack-tools-1.4.4-7.el7.x86_64.rpm
    ├── cri-tools-1.13.0-0.x86_64.rpm
    ├── kubeadm-1.19.0-0.x86_64.rpm
    ├── kubectl-1.19.0-0.x86_64.rpm
    ├── kubelet-1.19.0-0.x86_64.rpm
    ├── kubernetes-cni-0.8.7-0.x86_64.rpm
    ├── libnetfilter_cthelper-1.0.0-11.el7.x86_64.rpm
    ├── libnetfilter_cttimeout-1.0.0-7.el7.x86_64.rpm
    ├── libnetfilter_queue-1.0.2-2.el7_2.x86_64.rpm
    └── socat-1.7.3.2-2.el7.x86_64.rpm

6 directories, 32 files

k8s.tar

链接：https://pan.baidu.com/s/1_LgbKOc8VT6VFi4G1HVqtA
提取码：n4j2
复制这段内容后打开百度网盘手机App，操作更方便哦

以上是关于离线部署k8s-1.19.0集群的主要内容，如果未能解决你的问题，请参考以下文章

CDH5.16.1集群企业真正离线部署

Apache Hadoop集群离线安装部署——Hbase安装

K3S 离线安装部署高可用集群

Harbor高可用集群设计及部署（实操+视频），基于离线安装方式

Apache Hadoop集群离线安装部署——Hadoop(HDFSYARNMR)安装

离线部署 Cloudera Manager 5 和 CDH 5.12.1 及使用 CDH 部署 Hadoop 集群服务