Kubernetes安装手记-kubeadm

Posted 2022-08-27

十、Kubernetes实践篇

10.1) Kubernetes安装三种方式（官方提供）

10.1.1）minikube
Minikube是一个工具，可以在本地快速运行一个单点的Kubernetes，尝试Kubernetes或日常开发的用户使用。不能用于生产环境。

官方文档：

https://kubernetes.io/docs/setup/minikube/

10.1.2）kubeadm
kubeadm可帮助你快速部署一套kubernetes集群。kubeadm设计目的为新用户开始尝试kubernetes提供一种简单的方法。目前是Beta版。

官方文档：

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
https://kubernetes.io/docs/setup/independent/install-kubeadm/

10.1.3）二进制包
从官方下载发行版的二进制包，手动部署每个组件，组成Kubernetes集群。目前企业生产环境中主要使用该方式。

下载地址：

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1113

10.2) Kubernetes安装规划

10.2.1）基本资源
操作系统
Ubuntu 16.04+
Debian 9
CentOS 7
RHEL 7
Fedora 25/26 (best-effort)
其它要求：
内存2GB + ，2核CPU +（生产具体规划）
集群节点之间可以通信
每个节点唯一主机名，MAC地址和product_uuid
检查MAC地址：使用ip link或者ifconfig -a
检查product_uuid：cat /sys/class/dmi/id/product_uuid
禁止swap分区，这样才能使kubelet正常工作

10.2.2）节点规划

192.168.111.134 node7  --node1
192.168.111.135 node8  --node2
192.168.111.136 node9  --master

10.3) Kubernetes准备环境

10.3.1）关闭防火墙

 systemctl stop firewalld
 systemctl disable firewalld

10.3.2）关闭selinux

 sed -i ‘s/enforcing/disabled/‘ /etc/selinux/config 
 setenforce 0

10.3.3）关闭swap

swapoff -a  # 临时
swapoff -a && sysctl -w vm.swappiness=0
vim /etc/fstab  # 永久
sed -ri ‘/^[^#]*swap/s@^@#@‘ /etc/fstab

更改swap限制

 cat /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS=--fail-swap-on=false

10.3.4）hosts配置

192.168.111.134 node7
192.168.111.135 node8
192.168.111.136 node9

10.3.5）时间同步

ntpdate 1.cn.pool.ntp.org
yum install ntpdate –y

#配置ntp
client

Server：

10.3.6）添加ssh互信

ssh-keygen -t rsa
for i in node7 node8 node9;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
yum install -y bridge-utils.x86_64

10.4) Kubernetes集群安装（kubeadm）

10.4.1）系统资源参数

*   hardnofile  65536
*   softnofile  65536
*   hardnproc   65536
*   softnproc   65536

编辑配置文件/etc/sysctl.conf，添加以下内容：

net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 0
net.ipv4.ip_local_port_range = 10240 65535
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_keepalive_time = 1200
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.somaxconn = 16384

保存退出后执行sysctp -p生效

10.4.2）docker安装
2.6）Docker 安装管理
10.4.3）kubeadm相关工具安装
kubeadm： 引导集群的命令
kubelet：集群中运行任务的代理程序
kubectl：命令行管理工具
添加阿里云YUM软件源

#cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

或者：

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
 yum install -y kubelet kubeadm kubectl kubernetes-cni
yum install -y kubelet-1.13.5-0.x86_64 kubeadm-1.13.5-0.x86_64 kubectl-1.13.5-0.x86_64 kubernetes-cni 
 systemctl enable kubelet && systemctl start kubelet

注意：使用Docker时，kubeadm会自动检查kubelet的cgroup驱动程序，并/var/lib/kubelet/kubeadm-flags.env在运行时将其设置在文件中。如果使用的其他CRI，则必须在/etc/default/kubelet中cgroup-driver值修改为cgroupfs：

cat /var/lib/kubelet/kubeadm-flags.env
KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
systemctl daemon-reload
systemctl restart kubelet

kubeadm 常用的命令

 helpHelp about any command 
 initRun this command in order to set up the Kubernetes control plane. # master上执行，初始化所有的master组件
 joinRun this on any machine you wish to join an existing cluster # node上执行，加入master
 reset   Run this to revert any changes made to this host by ‘kubeadm init‘ or ‘kubeadm join‘. # 清理 init，join的环境
 token   Manage bootstrap tokens. # token的增删查
 upgrade Upgrade your cluster smoothly to a newer version with this command. # 更新集群
 version Print the version of kubeadm

10.4.4）下载kubernetes的相关镜像

K8S_VERSION=v1.13.5
ETCD_VERSION=3.2.24
DASHBOARD_VERSION=v1.8.3
FLANNEL_VERSION=v0.10.0-amd64
DNS_VERSION=1.2.6
PAUSE_VERSION=3.1

基本组件

docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$K8S_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:$ETCD_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$DNS_VERSION

网络组件

docker pull quay.io/coreos/flannel:$FLANNEL_VERSION

修改tag

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:$K8S_VERSION k8s.gcr.io/kube-apiserver:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:$K8S_VERSION k8s.gcr.io/kube-controller-manager:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:$K8S_VERSION k8s.gcr.io/kube-scheduler:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:$K8S_VERSION k8s.gcr.io/kube-proxy:$K8S_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:$ETCD_VERSION k8s.gcr.io/etcd:$ETCD_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:$PAUSE_VERSION k8s.gcr.io/pause:$PAUSE_VERSION
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:$DNS_VERSION k8s.gcr.io/coredns:$DNS_VERSION
10.4.5）运行kubeadm init 安装master

配置国内镜像加速

cat /etc/docker/daemon.json 

"registry-mirrors": ["https://registry.docker-cn.com" ]

kubeadm config

kubeadm config upload from-file 由配置文件上传到集群中生成ConfigMap
kubeadm config upload from-flags 由配置参数生成ConfigMap
kubeadm config view 查看当前集群中的配置值
kubeadm config print init-defaults 输出init-defaults默认参数文件内容
kubeadm config print join-defaults 输出join-defaults默认参数文件内容
kubeadm config migrate 在新旧版本之间进行配置转换
kubeadm config images list 列出所需镜像列表
kubeadm config images pull 拉去镜像到本地

新建init-config.yaml文件定制镜像仓库地址和Pod地址段
cat init-config.yaml

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
imageRepository: docker.io/dustise
kubernetesVersion: v1.14.0
networking:
  podSubnet: "172.16.0.0/16 "

下载所需镜像

kubeadm config images pull --config=init-config.yaml
[config/images] Pulled docker.io/dustise/kube-apiserver:v1.14.0
[config/images] Pulled docker.io/dustise/kube-controller-manager:v1.14.0
[config/images] Pulled docker.io/dustise/kube-scheduler:v1.14.0
[config/images] Pulled docker.io/dustise/kube-proxy:v1.14.0
[config/images] Pulled docker.io/dustise/pause:3.1
[config/images] Pulled docker.io/dustise/etcd:3.3.10
[config/images] Pulled docker.io/dustise/coredns:1.3.1

查看默认参数文件

kubeadm config print init-defaults

出现[WARNING IsDockerSystemdCheck]，是由于docker的Cgroup Driver和kubelet的Cgroup Driver不一致导致的，此处选择修改docker的和kubelet一致

docker info | grep Cgroup
Cgroup Driver: cgroupfs
编辑文件/usr/lib/systemd/system/docker.service

ExecStart=/usr/bin/dockerd --exec-opt native.cgroupdriver=systemd
systemctl daemon-reload
systemctl restart docker

docker info | grep Cgroup
Cgroup Driver: system

操作
修改配置

sed -e ‘s/KUBELET_CGROUP_ARGS=--cgroup-driver=systemd/KUBELET_CGROUP_ARGS=--cgroup-driver=cgroupfs/‘ /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

DOCKER_CGROUPS=$(docker info | grep ‘Cgroup‘ | cut -d‘ ‘ -f3)
echo $DOCKER_CGROUPS
echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables
kubeadm init --kubernetes-version=1.13.5 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.111.136

kebeadm init需要加上参数
详细的参数介绍可看：

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/

10.4.5）运行kubeadm init 安装master
配置国内镜像加速

cat /etc/docker/daemon.json 

"registry-mirrors": ["https://registry.docker-cn.com" ]

kubeadm config

kubeadm config upload from-file 由配置文件上传到集群中生成ConfigMap
kubeadm config upload from-flags 由配置参数生成ConfigMap
kubeadm config view 查看当前集群中的配置值
kubeadm config print init-defaults 输出init-defaults默认参数文件内容
kubeadm config print join-defaults 输出join-defaults默认参数文件内容
kubeadm config migrate 在新旧版本之间进行配置转换
kubeadm config images list 列出所需镜像列表
kubeadm config images pull 拉去镜像到本地

新建init-config.yaml文件定制镜像仓库地址和Pod地址段

#cat init-config.yaml 
apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
imageRepository: docker.io/dustise
kubernetesVersion: v1.14.0
networking:
  podSubnet: "172.16.0.0/16 "

下载所需镜像

kubeadm config images pull --config=init-config.yaml
[config/images] Pulled docker.io/dustise/kube-apiserver:v1.14.0
[config/images] Pulled docker.io/dustise/kube-controller-manager:v1.14.0
[config/images] Pulled docker.io/dustise/kube-scheduler:v1.14.0
[config/images] Pulled docker.io/dustise/kube-proxy:v1.14.0
[config/images] Pulled docker.io/dustise/pause:3.1
[config/images] Pulled docker.io/dustise/etcd:3.3.10
[config/images] Pulled docker.io/dustise/coredns:1.3.1

#
DOCKER_CGROUPS=$(docker info | grep ‘Cgroup‘ | cut -d‘ ‘ -f3)
echo $DOCKER_CGROUPS
echo 1 >/proc/sys/net/bridge/bridge-nf-call-iptables
kubeadm init --kubernetes-version=1.13.5 --pod-network-cidr=172.16.0.0/16 --apiserver-advertise-address=192.168.111.136

kebeadm init需要加上参数 详细的参数介绍可看：

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/

使用kubeadm reset重置主机状态然后重新初始化

10.4.6）常规用户使用kubectl访问集群

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

10.4.7）安装pod网络插件Flannel

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

安装网络插件weave

kubectl apply -f https://cloud.weave.works/k8s/net?k8s-version=$(kubectl version | base64 | tr -d ‘\n‘)

10.4.8）查看所有pod和节点

10.4.9）添加工作节点
在Node节点切换到root账号执行：

kubeadm join 192.168.111.136:6443 --token gf25fd.xntkm8qy5klmhrv6 --discovery-token-ca-cert-hash sha256:f409b76900e0bf4e334f1bc2b629a89f4e031744489c6bfe8d8233f9af7ecdd7
#格式：kubeadm join --token <token> <master-ip>:<master-port> --discovery-token-ca-cert-hash sha256:<hash>

10.4.10）安装配置访问dashboardb
安装dashboard

https://github.com/kubernetes/dashboard
wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml
wget https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.0/src/deploy/recommended/kubernetes-dashboard.yaml

修改Dashboard Service 为NodePort类型

kind: Service
apiVersion: v1
metadata:
  labels:
k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  type: NodePort
  ports:
- port: 443
  targetPort: 8443
  nodePort: 30001
  selector:
k8s-app: kubernetes-dashboard

进行部署

kubectl create -f kubernetes-dashboard.yaml
kubectl delete -f kubernetes-dashboard.yaml

查验

kubectl get svc --all-namespaces

创建管理员

cat k8s-admin.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
k8s-app: kubernetes-dashboard
  name: admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin
  namespace: kube-system

使用token登录

https://192.168.111.136:30001/#!/login