k8s集群环境搭建
Posted 爱上口袋的天空
tags:
篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了k8s集群环境搭建相关的知识，希望对你有一定的参考价值。
1.环境准备
1.1）机器环境
节点CPU核数必须是 ：>= 2核 ，否则k8s无法启动 DNS网络： 最好设置为 本地网络连通的DNS,否则网络不通，无法下载一些镜像 linux内核： linux内核必须是 4 版本以上，
因此必须把linux核心进行升级
准备3台虚拟机环境，或者是3台阿里云服务器都可。
centos002: 此机器用来安装k8s-master的操作环境
centos003: 此机器用来安装k8s node节点的环境
centos004: 此机器用来安装k8s node节点的环境
使用命令查看Linux内核版本：
由上面可以看出来我们的Linux内核后面需要升级为4版本以上，目前是3版本不满足要求。1.2）设置3台机器的主机名，我们这里就叫做centos002,centos003,centos004
1.3) 配置IP host映射关系,3台机器都要配置
1.4) 安装依赖环境，注意：每一台机器都需要安装此依赖环境
yum install -y conntrack ntpdate ntp ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git iproute lrzsz bash-completion tree bridge-utils unzip bind-utils gcc
1.5）关闭防火墙，每一台机器都需要
命令：systemctl stop firewalld && systemctl disable firewalld
1.6）安装iptables，启动iptables，设置开机自启，清空iptables规则，保存当前规则到默认规则，3台机器都要配置
命令：yum -y install iptables-services && systemctl start iptables && systemctl enable iptables && iptables -F && service iptables save
1.7）关闭swap分区【虚拟内存】并且永久关闭虚拟内存，3台机器都要配置
命令：swapoff -a && sed -i '/ swap / s/^\\(.*\\)$/#\\1/g' /etc/fstab
1.8）关闭selinux，3台机器都要配置
命令：setenforce 0 && sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config
1.9）升级Linux内核为5.4版本，3台机器都要配置
命令：rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
安装内核：yum --enablerepo=elrepo-kernel install -y kernel-lt
设置开机从新内核启动：grub2-set-default 'CentOS Linux (5.4.117-1.el7.elrepo.x86_64) 7 (Core)'
注意：设置完内核后，需要重启服务器才会生效
1.10）调整内核参数，对于k8s，3台机器都要配置
在/opt目录下创建kubernetes.conf文件，内容如下：
net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 net.ipv4.tcp_tw_recycle=0 vm.swappiness=0 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_instances=8192 fs.inotify.max_user_watches=1048576 fs.file-max=52706963 fs.nr_open=52706963 net.ipv6.conf.all.disable_ipv6=1 net.netfilter.nf_conntrack_max=2310720将优化内核文件拷贝到/etc/sysctl.d/文件夹下，这样优化文件开机的时候能够被调用
命令：cp /opt/kubernetes.conf /etc/sysctl.d/kubernetes.conf手动刷新，让优化文件立即生效:
命令：sysctl -p /etc/sysctl.d/kubernetes.conf
1.11）调整系统临时区 --- 如果已经设置时区，可略过，3台机器都要配置
1）设置系统时区为中国/上海
命令：timedatectl set-timezone Asia/Shanghai2）将当前的 UTC 时间写入硬件时钟
命令：timedatectl set-local-rtc 03）重启依赖于系统时间的服务
命令：
systemctl restart rsyslog
systemctl restart crond1.12）关闭系统不需要的服务，3台机器都要配置
命令：systemctl stop postfix && systemctl disable postfix1.13）设置日志保存方式，3台机器都要配置
1）创建保存日志的目录
命令：mkdir /var/log/journal2）创建配置文件存放目录
命令：mkdir /etc/systemd/journald.conf.d3）在/etc/systemd/journald.conf.d目录下创建配置文件99-prophet.conf
[Journal] Storage=persistent #表示存储日志是否需要持久化 Compress=yes #表示日志是否需要压缩 SyncIntervalSec=5m #同步的时间间隔为5分钟 RateLimitInterval=30s #频率的限制是30秒 RateLimitBurst=1000 SystemMaxUse=10G #系统使用的空间限制10G SystemMaxFileSize=200M #系统最大文件大小 MaxRetentionSec=2week ForwardToSyslog=no4） 重启systemd journald的配置，这个是专门用来收集日志的
命令：systemctl restart systemd-journald5）打开文件数调整 (可忽略，不执行)
命令：
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf1.14）kube-proxy 开启 ipvs 前置条件，3台机器都要配置
命令：modprobe br_netfilter
在/etc/sysconfig/modules/目录下创建ipvs.modules文件#!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack使用lsmod命令查看这些文件是否被引导
命令：chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack
2.docker部署，3台机器都要配置
- 安装docker
命令：yum install -y yum-utils device-mapper-persistent-data lvm2- 紧接着配置一个稳定（stable）的仓库、仓库配置会保存到/etc/yum.repos.d/docker-ce.repo文件 中
命令：yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo- 更新Yum安装的相关Docke软件包&安装Docker CE
命令：yum update -y && yum install -y docker-ce- 创建/etc/docker目录
命令：mkdir /etc/docker- 更新daemon.json文件，在/etc/docker/目录下创建daemon.json文件，内容如下
{"exec-opts": ["native.cgroupdriver=systemd"],"log-driver": "json-file","log-opts": {"max-size": "100m"}}
- 创建，存储docker配置文件
命令：mkdir -p /etc/systemd/system/docker.service.d- 重启docker服务
命令：systemctl daemon-reload && systemctl restart docker && systemctl enable docker- 查看docker信息
3.kubeadm[一键安装k8s]，3台机器都要配置
- 安装kubernetes的时候，需要安装kubelet, kubeadm等包，但k8s官网给的yum源是 packages.cloud.google.com，国内访问不了，此时我们可以使用阿里云的yum仓库镜像。
创建/etc/yum.repos.d/kubernetes.repo文件，内容如下[kubernetes] name=Kubernetes baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
- 安装kubeadm、kubelet、kubectl
命令：yum install -y kubeadm-1.15.1 kubelet-1.15.1 kubectl-1.15.1- 启动 kubelet
命令：systemctl enable kubelet && systemctl start kubelet
4.集群安装
- 将基础镜像上传到centos002服务器上
- 解压安装包
- 编写脚本问题，导入镜像包到本地docker镜像仓库：
# kubeadm 初始化k8s集群的时候，会从gce Google云中下载（pull）相应的镜像,且镜像相对比较大，
下载比较慢，且需要解决科学上网的一个问题，国内上goole，懂得...........
1）在/opt目录下创建image-load.sh，内容如下
#!/bin/bash #注意 镜像解压的目录位置 ls /opt/kubeadm-basic.images > /tmp/images-list.txt cd /opt/kubeadm-basic.images for i in $(cat /tmp/images-list.txt) do docker load -i $i done rm -rf /tmp/images-list.txt
2）修改权限，可执行权限
命令：chmod 755 image-load.sh
3）开始执行,镜像导入
4）传输文件及镜像到其他node节点
#拷贝到node01节点
scp -r image-load.sh kubeadm-basic.images root@centos003:/opt/
#拷贝到node02节点
scp -r image-load.sh kubeadm-basic.images root@centos004:/opt/
#其他节点依次执行sh脚本，导入镜像
5）导入成功后镜像仓库如下图所示：
- k8s部署
1）初始化主节点 --- 只需要在主节点centos002执行
1.1）拉去yaml资源配置文件
命令：kubeadm config print init-defaults > kubeadm-config.yaml
1.2）修改yaml资源文件,最终内容如下
apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: #注意：修改配置文件的IP地址 advertiseAddress: 192.168.56.11 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: centos002 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration #注意：修改版本号，必须和kubectl版本保持一致 kubernetesVersion: v1.15.1 networking: #指定flannel模型通信 pod网段地址,此网段和flannel网段一致 podSubnet: 10.244.0.0/16 dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {} #指定使用ipvs网络进行通信 --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: kubeProxyConfiguration featureGates: SupportIPVSProxyMode: true mode: ipvs
1.3) 初始化主节点，开始部署，#注意：执行此命令，CPU核心数量必须大于1核，否则无法执行成功
命令：kubeadm init --config=kubeadm-config.yaml --experimental-upload-certs | tee kubeadm-init.log
1.4）按照k8s指示，执行下面的命令：
#创建目录，保存连接配置缓存，认证文件：mkdir -p $HOME/.kube
#拷贝集群管理配置文件 ： cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
#授权给配置文件 ：chown $(id -u):$(id -g) $HOME/.kube/confi
1.5）执行命令前查询node:
命令：kubectl get node
我们发现已经可以成功查询node节点信息了，但是节点的状态却是NotReady,不是Runing的状态。原 因是此时我们使用ipvs+flannel的方式进行网络通信，
但是flannel网络插件还没有部署，因此节点状态 此时为NotReady
2）部署flannel网络插件 --- 只需要在主节点执行
2.1)下载flannel网络插件
命令：wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
如果下载不下来直接只用下来的即可， kube-flannel.yml内容如下：
--- apiVersion: policy/v1beta1 kind: PodSecurityPolicy metadata: name: psp.flannel.unprivileged annotations: seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default spec: privileged: false volumes: - configMap - secret - emptyDir - hostPath allowedHostPaths: - pathPrefix: "/etc/cni/net.d" - pathPrefix: "/etc/kube-flannel" - pathPrefix: "/run/flannel" readOnlyRootFilesystem: false # Users and groups runAsUser: rule: RunAsAny supplementalGroups: rule: RunAsAny fsGroup: rule: RunAsAny # Privilege Escalation allowPrivilegeEscalation: false defaultAllowPrivilegeEscalation: false # Capabilities allowedCapabilities: ['NET_ADMIN'] defaultAddCapabilities: [] requiredDropCapabilities: [] # Host namespaces hostPID: false hostIPC: false hostNetwork: true hostPorts: - min: 0 max: 65535 # SELinux seLinux: # SELinux is unused in CaaSP rule: 'RunAsAny' --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: flannel rules: - apiGroups: ['extensions'] resources: ['podsecuritypolicies'] verbs: ['use'] resourceNames: ['psp.flannel.unprivileged'] - apiGroups: - "" resources: - pods verbs: - get - apiGroups: - "" resources: - nodes verbs: - list - watch - apiGroups: - "" resources: - nodes/status verbs: - patch --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: flannel roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: flannel subjects: - kind: ServiceAccount name: flannel namespace: kube-system --- apiVersion: v1 kind: ServiceAccount metadata: name: flannel namespace: kube-system --- kind: ConfigMap apiVersion: v1 metadata: name: kube-flannel-cfg namespace: kube-system labels: tier: node app: flannel data: cni-conf.json: | { "name": "cbr0", "cniVersion": "0.3.1", "plugins": [ { "type": "flannel", "delegate": { "hairpinMode": true, "isDefaultGateway": true } }, { "type": "portmap", "capabilities": { "portMappings": true } } ] } net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "vxlan" } } --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-amd64 namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/os operator: In values: - linux - key: beta.kubernetes.io/arch operator: In values: - amd64 hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-amd64 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-amd64 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-arm64 namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/os operator: In values: - linux - key: beta.kubernetes.io/arch operator: In values: - arm64 hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-arm64 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-arm64 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-arm namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/os operator: In values: - linux - key: beta.kubernetes.io/arch operator: In values: - arm hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-arm command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-arm command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-ppc64le namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/os operator: In values: - linux - key: beta.kubernetes.io/arch operator: In values: - ppc64le hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-ppc64le command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-ppc64le command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-s390x namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/os operator: In values: - linux - key: beta.kubernetes.io/arch operator: In values: - s390x hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-s390x command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay-mirror.qiniu.com/coreos/flannel:v0.12.0-s390x command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN"] env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace volumeMounts: - name: run mountPath: /run/flannel - name: flannel-cfg mountPath: /etc/kube-flannel/ volumes: - name: run hostPath: path: /run/flannel - name: cni hostPath: path: /etc/cni/net.d - name: flannel-cfg configMap: name: kube-flannel-cfg
2.2)部署flannel
命令：kubectl create -f kube-flannel.yml
2.3）使用命令查看组件状态
命令：kubectl get pod -n kube-system
发现通过flannel部署的pod都出现pending,ImagePullBackOff这样的问题：
查询指定pod的详细错误：kubectl describe pod kube-flannel-ds-amd64-x2jhv -n kube-system
解决办法参考：https://blog.csdn.net/K_520_W/article/details/116566733
2.4）最后结果
3）节点Join
构建kubernetes主节点成功，会产生一个日志文件（命令中指定日志输出文件 “tee kubeadm-init.log”），内容如下所示
负责命令到其他几个node（centos003,centos004）节点进行执行即可：
执行完毕，查看效果如下所示：
出现上面的效果表示我们的k8s集群已经搭建成功
以上是关于k8s集群环境搭建的主要内容，如果未能解决你的问题，请参考以下文章