华为云+阿里云 不同云服务器部署Kubernetes(K8S)

Posted 春茶

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了华为云+阿里云 不同云服务器部署Kubernetes(K8S)相关的知识，希望对你有一定的参考价值。

前言

经历了一周的高强度部署，踩了无数的坑后终于搭起了华为云+阿里云的集群，十分感觉@chen645800876大佬的云服务器-异地部署集群服务这篇文章，才能比较顺利的部署，少踩了很多坑。这次记录是基于大佬文章上，减少了一些我没有使用的步骤，也把我踩的坑记录一下，做一个备份也希望能帮助到其他人。

正式安装

  1. 调整内核参数

    cat > k8s.conf <<EOF
#开启网桥模式
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
#开启转发
net.ipv4.ip_forward = 1
##关闭ipv6
net.ipv6.conf.all.disable_ipv6=1
EOF
cp k8s.conf /etc/sysctl.d/k8s.conf
sysctl -p /etc/sysctl.d/k8s.conf

  2. ipvs前置条件准备

    # step1
modprobe br_netfilter

# step2
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
EOF

# step3
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack

    这个地方需要注意一下的是原文中模块nf_conntrack_ipv4已经没有使用了，解决方法是下面链接提出的方案，这个地方非常重要，如果抛错的话，后面ipvs转发会有问题
    https://github.com/easzlab/ku...

  3. 关闭swap分区

    swapoff -a

  4. Kubeadm、Kubelet、Kubectl安装

    # 添加源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
#baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-aarch64/
#如果是处理器不是amd的话就需要用到另外一个版本 华为鲲鹏型的就是aarch64 而阿里的是x86_64
#这个还有个新手小坑，就是docker的镜像也跟处理器版本有关。x86_64上打的包，aarch64的docker就不能发布，如果遇到pod发布不成功有可能是这个问题
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 关闭selinux
setenforce 0

# 安装kubelet、kubeadm、kubectl
yum install -y kubelet kubeadm kubectl

# 设置为开机自启
systemctl enable kubelet

  5. 建立虚拟网卡

    # step1 ，注意替换你的公网IP进去
cat > /etc/sysconfig/network-scripts/ifcfg-eth0:1 <<EOF
BOOTPROTO=static
DEVICE=eth0:1
IPADDR=你的公网IP
PREFIX=32
TYPE=Ethernet
USERCTL=no
ONBOOT=yes
EOF
# step2 如果是centos8，需要重启）（建议直接换成centos7，centos8的网卡设置复杂一些）
# 华为云服务器在网卡设置上是默认了有eth1-eth5 所以需要把默认的这些全部取消 不然会抛错导致网卡无法重启
systemctl restart network
# step3 查看新建的IP是否进去
ip addr

  6. 修改kubelet启动参数（重点，所有节点都要操作）

    # 此文件安装kubeadm后就存在了
vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf

# 注意，这步很重要，如果不做，节点仍然会使用内网IP注册进集群
# 在末尾添加参数 --node-ip=公网IP

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip=xx.xx.xx.xx

  7. 使用kubeadm初始化主节点,提供两个脚本可以下载和清理镜像

    #! /bin/bash
images=(
 kube-apiserver:v1.21.1
 kube-controller-manager:v1.21.1
 kube-scheduler:v1.21.1
 kube-proxy:v1.21.1
 pause:3.4.1
 etcd:3.4.13-0
 #coredns/coredns 直接从dockerhub上下载
)
for imageName in ${images[@]} ; do
 docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/${imageName}
 docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/${imageName} k8s.gcr.io/${imageName}
 docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/${imageName}
done

    #! /bin/bash

images=`docker images|grep k8s.gcr|awk \'{print $3}\'`
for image in ${images}
do
echo $image
docker rmi $image
done
    # step1 添加配置文件，注意替换下面的IP
cat > kubeadm-config.yaml <<EOF
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.21.1
apiServer:
  certSANs:    #填写所有kube-apiserver节点的hostname、IP、VIP
  - master    #请替换为hostname
  - xx.xx.xx.xx   #请替换为公网
  - yy.yy.yy.yy  #请替换为私网
  - 10.96.0.1   #不要替换，此IP是API的集群地址，部分服务会用到
controlPlaneEndpoint: xx.xx.xx.xx:6443 #替换为公网IP
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
--- 将默认调度方式改为ipvs
apiVersion: kubeproxy-config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
featureGates:
  SupportIPVSProxyMode: true
mode: ipvs
EOF

# step2 如果是1核心或者1G内存的请在末尾添加参数（--ignore-preflight-errors=all），否则会初始化失败
# 同时注意，此步骤成功后，会打印，两个重要信息
kubeadm init --config=kubeadm-config.yaml 

# 信息1 上面初始化成功后，将会生成kubeconfig文件，用于请求api服务器，请执行下面操作
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 信息2 此信息用于后面工作节点加入主节点使用
kubeadm join xx.xx.xx.xx:6443 --token sdfs.dsfsdfsdfijdth \\
 --discovery-token-ca-cert-hash sha256:sdfsdfsdfsdfsdfsdfsdfsdfg9a460f44b118050091245c1d

  1. 修改kube-apiserver参数（主节点）

    # 修改三个信息，添加--bind-address和修改--advertise-address和feature-gates=RemoveSelfLink
vim /etc/kubernetes/manifests/kube-apiserver.yaml

apiVersion: v1
kind: Pod
metadata:
  annotations:
 kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: 47.74.22.13:6443
  creationTimestamp: null
  labels:
 component: kube-apiserver
 tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
 - kube-apiserver
 - --feature-gates=RemoveSelfLink=false  #如果涉及到NFS挂载StorageClass需要增加这个参数 k8s 1.20后就取消了这个参数所以需要手动增加
#解决方法是来源于https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner/issues/25
 - --advertise-address=47.74.22.13  #修改为公网IP
 - --bind-address=0.0.0.0 #添加此参数
 - --allow-privileged=true
 - --authorization-mode=Node,RBAC
 - --client-ca-file=/etc/kubernetes/pki/ca.crt
 - --enable-admission-plugins=NodeRestriction
 - --enable-bootstrap-token-auth=true
 - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
 - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
 - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
 - --etcd-servers=https://127.0.0.1:2379
 - --insecure-port=0
 - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
 - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
 - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
 - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
 - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
 - --requestheader-allowed-names=front-proxy-client
 - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
 - --requestheader-extra-headers-prefix=X-Remote-Extra-
 - --requestheader-group-headers=X-Remote-Group
 - --requestheader-username-headers=X-Remote-User
 - --secure-port=6443
 - --service-account-key-file=/etc/kubernetes/pki/sa.pub
 - --service-cluster-ip-range=10.96.0.0/12
 - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
 - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
 image: k8s.gcr.io/kube-apiserver:v1.18.0
 imagePullPolicy: IfNotPresent
 livenessProbe:
   failureThreshold: 8
   httpGet:
     host: 175.24.19.12
     path: /healthz
     port: 6443
     scheme: HTTPS
   initialDelaySeconds: 15
   timeoutSeconds: 15
 name: kube-apiserver
 resources:
   requests:
     cpu: 250m
 volumeMounts:
 - mountPath: /etc/ssl/certs
   name: ca-certs
   readOnly: true
 - mountPath: /etc/pki
   name: etc-pki
   readOnly: true
 - mountPath: /etc/kubernetes/pki
   name: k8s-certs
   readOnly: true
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
   path: /etc/ssl/certs
   type: DirectoryOrCreate
 name: ca-certs
  - hostPath:
   path: /etc/pki
   type: DirectoryOrCreate
 name: etc-pki
  - hostPath:
   path: /etc/kubernetes/pki
   type: DirectoryOrCreate
 name: k8s-certs
status: {}

  2. 修改flannel文件并安装（主节点）

    wget https://raw.githubusercontent.com/coreos/flanne/master/Documentation/kube-flannel.yml


apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-flannel-ds-amd64
  namespace: kube-system
  labels:
 tier: node
 app: flannel
spec:
  selector:
 matchLabels:
   app: flannel
  template:
 metadata:
   labels:
     tier: node
     app: flannel
 spec:
   affinity:
     nodeAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
         nodeSelectorTerms:
           - matchExpressions:
               - key: beta.kubernetes.io/os
                 operator: In
                 values:
                   - linux
               - key: beta.kubernetes.io/arch
                 operator: In
                 values:
                   - amd64
   hostNetwork: true
   tolerations:
   - operator: Exists
     effect: NoSchedule
   serviceAccountName: flannel
   initContainers:
   - name: install-cni
     image: quay.io/coreos/flannel:v0.11.0-amd64
     command:
     - cp
     args:
     - -f
     - /etc/kube-flannel/cni-conf.json
     - /etc/cni/net.d/10-flannel.conflist
     volumeMounts:
     - name: cni
       mountPath: /etc/cni/net.d
     - name: flannel-cfg
       mountPath: /etc/kube-flannel/
   containers:
   - name: kube-flannel
     image: quay.io/coreos/flannel:v0.11.0-amd64
     command:
     - /opt/bin/flanneld
     args:
     - --ip-masq
     - --public-ip=$(PUBLIC_IP) # 添加此参数，申明公网IP
     - --iface=eth0             # 添加此参数，绑定网卡
     - --kube-subnet-mgr
     resources:
       requests:
         cpu: "100m"
         memory: "50Mi"
       limits:
         cpu: "100m"
         memory: "50Mi"
     securityContext:
       privileged: false
       capabilities:
          add: ["NET_ADMIN"]
     env:
     - name: PUBLIC_IP     #添加环境变量
       valueFrom:          #
         fieldRef:          #
           fieldPath: status.podIP # 
     - name: POD_NAME
       valueFrom:
         fieldRef:
           fieldPath: metadata.name

  3. 手动开启配置，开启ipvs转发模式（主节点）

    # 前面都成功了，但是有时候默认并不会启用`IPVS`模式，那就手动修改一下，只修改一处
# 修改后，如果没有及时生效，请删除kube-proxy，会自动重新创建，然后使用ipvsadm -Ln命令，查看是否生效
# ipvsadm没有安装的，使用yum install ipvsadm安装
kubectl edit configmaps -n kube-system kube-proxy

---
apiVersion: v1
data:
  config.conf: |-
 apiVersion: kubeproxy.config.k8s.io/v1alpha1
 bindAddress: 0.0.0.0
 clientConnection:
   acceptContentTypes: ""
   burst: 0
   contentType: ""
   kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
   qps: 0
 clusterCIDR: 10.244.0.0/16
 configSyncPeriod: 0s
 conntrack:
   maxPerCore: null
   min: null
   tcpCloseWaitTimeout: null
   tcpEstablishedTimeout: null
 detectLocalMode: ""
 enableProfiling: false
 healthzBindAddress: ""
 hostnameOverride: ""
 iptables:
   masqueradeAll: false
   masqueradeBit: null
   minSyncPeriod: 0s
   syncPeriod: 0s
 ipvs:
   excludeCIDRs: null
   minSyncPeriod: 0s
   scheduler: ""
   strictARP: false
   syncPeriod: 0s
   tcpFinTimeout: 0s
   tcpTimeout: 0s
   udpTimeout: 0s
 kind: KubeProxyConfiguration
 metricsBindAddress: ""
 mode: "ipvs"  # 如果为空，请填入`ipvs`
 nodePortAddresses: null
 oomScoreAdj: null
 portRange: ""
 showHiddenMetricsForVersion: ""
 udpIdleTimeout: 0s
 winkernel:
   enableDSR: false
   networkName: ""

以上是关于华为云+阿里云 不同云服务器部署Kubernetes(K8S)的主要内容，如果未能解决你的问题，请参考以下文章

用友云市场和阿里云市场华为云市场有何区别？

阿里云VS腾讯云 VS 华为云 国内三大云服务商云服务器对比

阿里云VS腾讯云 VS 华为云 国内三大云服务商云服务器对比

阿里云VS腾讯云 VS 华为云 国内三大云服务商云服务器对比

阿里云VS腾讯云 VS 华为云 国内三大云服务商云服务器对比

阿里云服务器是如何实现每台服务器都是公网IP的呢？