一个全新的基于容器技术的分布式架构方案-Kubernetes

Posted 程序员基础课

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了一个全新的基于容器技术的分布式架构方案-Kubernetes相关的知识,希望对你有一定的参考价值。

文末搜集了几张非常不错的架构图,从k8s基础组件,网络,工作流以及工作模式四个方面介绍k8s的工作模式,对深入理解k8s有很好的帮助。(文章较长,建议先收藏)

k8s(1.14.0)+etcd(3.3.10)+flanneld(0.10)

安装和配置

前提

os:centos 7.5.1804

一、初始化环境

1.1 关闭防火墙和SeLinux
 
   
   
 
  1. systemctl stop firewalld

  2. setenforce 0 (临时关闭)

  3. vi /etc/selinux/config

  4. SELINUX=disabled

1.2 下载链接
 
   
   
 
  1. Client Binaries

  2. https://dl.k8s.io/v1.14.0/kubernetes-client-linux-amd64.tar.gz

  3. Server Binaries

  4. https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz

  5. Node Binaries

  6. https://dl.k8s.io/v1.14.0/kubernetes-node-linux-amd64.tar.gz

  7. etcd

  8. https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz

  9. flannel

  10. https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

二、 k8s-Master部署

2.1 下载资源(wget不到的话,可以用科学上网的机器直接download)
 
   
   
 
  1. wget https://dl.k8s.io/v1.14.0/kubernetes-server-linux-amd64.tar.gz

  2. wget https://dl.k8s.io/v1.14.0/kubernetes-client-linux-amd64.tar.gz

  3. wget https://github.com/etcd-io/etcd/releases/download/v3.3.10/etcd-v3.3.10-linux-amd64.tar.gz

  4. wget https://github.com/coreos/flannel/releases/download/v0.10.0/flannel-v0.10.0-linux-amd64.tar.gz

2.2 cfssl安装(wget不到的话,可以用科学上网的机器直接download)
 
   
   
 
  1. wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

  2. wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

  3. wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

  4. chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

  5. mv cfssl_linux-amd64 /usr/local/bin/cfssl

  6. mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

  7. mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

2.3 创建etcd证书
 
   
   
 
  1. mkdir /k8s/etcd/{bin,cfg,ssl} -p

  2. mkdir /k8s/kubernetes/{bin,cfg,ssl} -p

  3. cd /k8s/etcd/ssl/

  • 2.3.1 etcd ca配置

 
   
   
 
  1. cat << EOF | tee ca-config.json

  2. {

  3. "signing": {

  4. "default": {

  5. "expiry": "87600h"

  6. },

  7. "profiles": {

  8. "etcd": {

  9. "expiry": "87600h",

  10. "usages": [

  11. "signing",

  12. "key encipherment",

  13. "server auth",

  14. "client auth"

  15. ]

  16. }

  17. }

  18. }

  19. }

  20. EOF

  • 2.3.2 etcd ca证书

 
   
   
 
  1. cat << EOF | tee ca-csr.json

  2. {

  3. "CN": "etcd CA",

  4. "key": {

  5. "algo": "rsa",

  6. "size": 2048

  7. },

  8. "names": [

  9. {

  10. "C": "CN",

  11. "L": "Beijing",

  12. "ST": "Beijing"

  13. }

  14. ]

  15. }

  16. EOF

  • 2.3.3 etcd server证书

 
   
   
 
  1. cat << EOF | tee server-csr.json

  2. {

  3. "CN": "etcd",

  4. "hosts": [

  5. "192.168.4.71",

  6. "192.168.4.72",

  7. "192.168.4.76"

  8. ],

  9. "key": {

  10. "algo": "rsa",

  11. "size": 2048

  12. },

  13. "names": [

  14. {

  15. "C": "CN",

  16. "L": "Beijing",

  17. "ST": "Beijing"

  18. }

  19. ]

  20. }

  21. EOF

  • 2.3.4 生成etcd ca证书和私钥 初始化ca

 
   
   
 
  1. cfssl gencert -initca ca-csr.json | cfssljson -bare ca

  • 2.3.5 生成server证书

 
   
   
 
  1. cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server

一个全新的基于容器技术的分布式架构方案-Kubernetes

2.4 etcd安装
  • 2.4.1 解压

 
   
   
 
  1. tar -xvf etcd-v3.3.10-linux-amd64.tar.gz

  2. cd etcd-v3.3.10-linux-amd64/

  3. cp etcd etcdctl /k8s/etcd/bin/

  • 2.4.2 配置etcd的配置文件

 
   
   
 
  1. [root@t71 cfg]# vim /k8s/etcd/cfg/etcd.conf


  2. #[Member]

  3. ETCD_NAME="etcd01"

  4. ETCD_DATA_DIR="/data1/etcd"

  5. ETCD_LISTEN_PEER_URLS="https://192.168.4.71:2380"

  6. ETCD_LISTEN_CLIENT_URLS="https://192.168.4.71:2379"


  7. #[Clustering]

  8. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.4.71:2380"

  9. ETCD_ADVERTISE_CLIENT_URLS="https://192.168.4.71:2379"

  10. ETCD_INITIAL_CLUSTER="etcd01=https://192.168.4.71:2380"

  11. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  12. ETCD_INITIAL_CLUSTER_STATE="new"


  13. #[Security]

  14. ETCD_CERT_FILE="/k8s/etcd/ssl/server.pem"

  15. ETCD_KEY_FILE="/k8s/etcd/ssl/server-key.pem"

  16. ETCD_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"

  17. ETCD_CLIENT_CERT_AUTH="true"

  18. ETCD_PEER_CERT_FILE="/k8s/etcd/ssl/server.pem"

  19. ETCD_PEER_KEY_FILE="/k8s/etcd/ssl/server-key.pem"

  20. ETCD_PEER_TRUSTED_CA_FILE="/k8s/etcd/ssl/ca.pem"

  21. ETCD_PEER_CLIENT_CERT_AUTH="true"

  • 2.4.3 配置etcd的启动文件

 
   
   
 
  1. mkdir /data1/etcd

  2. [root@t71 cfg]# vim /usr/lib/systemd/system/etcd.service


  3. [Unit]

  4. Description=Etcd Server

  5. After=network.target

  6. After=network-online.target

  7. Wants=network-online.target


  8. [Service]

  9. Type=notify

  10. WorkingDirectory=/data1/etcd/

  11. EnvironmentFile=-/k8s/etcd/cfg/etcd.conf

  12. # set GOMAXPROCS to number of processors

  13. ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /k8s/etcd/bin/etcd --name="${ETCD_NAME}" --data-dir="${ETCD_DATA_DIR}" --listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" --listen-peer-urls="${ETCD_LISTEN_PEER_URLS}" --advertise-client-urls="${ETCD_ADVERTISE_CLIENT_URLS}" --initial-cluster-token="${ETCD_INITIAL_CLUSTER_TOKEN}" --initial-cluster="${ETCD_INITIAL_CLUSTER}" --initial-cluster-state="${ETCD_INITIAL_CLUSTER_STATE}" --cert-file="${ETCD_CERT_FILE}" --key-file="${ETCD_KEY_FILE}" --trusted-ca-file="${ETCD_TRUSTED_CA_FILE}" --client-cert-auth="${ETCD_CLIENT_CERT_AUTH}" --peer-cert-file="${ETCD_PEER_CERT_FILE}" --peer-key-file="${ETCD_PEER_KEY_FILE}" --peer-trusted-ca-file="${ETCD_PEER_TRUSTED_CA_FILE}" --peer-client-cert-auth="${ETCD_PEER_CLIENT_CERT_AUTH}""

  14. Restart=on-failure

  15. LimitNOFILE=65536


  16. [Install]

  17. WantedBy=multi-user.targe

  • 2.4.4 启动

 
   
   
 
  1. systemctl daemon-reload

  2. systemctl enabel etcd

  3. systemctl start etcd

  • 2.4.5 检查服务

 
   
   
 
  1. [root@t71 bin]# etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://192.168.4.71:2379" cluster-health

  2. member ac829673d2b22824 is healthy: got healthy result from https://192.168.4.71:2379

  3. cluster is healthy

  4. [root@t71 bin]#

2.5 生成kubernets证书与私钥
  • 2.5.1 制作kubernetes ca证书

 
   
   
 
  1. cd /k8s/kubernetes/ssl

  2. cat << EOF | tee ca-config.json

  3. {

  4. "signing": {

  5. "default": {

  6. "expiry": "87600h"

  7. },

  8. "profiles": {

  9. "kubernetes": {

  10. "expiry": "87600h",

  11. "usages": [

  12. "signing",

  13. "key encipherment",

  14. "server auth",

  15. "client auth"

  16. ]

  17. }

  18. }

  19. }

  20. }

  21. EOF

 
   
   
 
  1. cat << EOF | tee ca-csr.json

  2. {

  3. "CN": "kubernetes",

  4. "key": {

  5. "algo": "rsa",

  6. "size": 2048

  7. },

  8. "names": [

  9. {

  10. "C": "CN",

  11. "L": "Beijing",

  12. "ST": "Beijing",

  13. "O": "k8s",

  14. "OU": "System"

  15. }

  16. ]

  17. }

  18. EOF

 
   
   
 
  1. cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

  • 2.5.2 制作apiserver证书

 
   
   
 
  1. cat << EOF | tee server-csr.json

  2. {

  3. "CN": "kubernetes",

  4. "hosts": [

  5. "10.254.0.1",

  6. "127.0.0.1",

  7. "192.168.4.71",

  8. "192.168.4.72",

  9. "192.168.4.76",

  10. "kubernetes",

  11. "kubernetes.default",

  12. "kubernetes.default.svc",

  13. "kubernetes.default.svc.cluster",

  14. "kubernetes.default.svc.cluster.local"

  15. ],

  16. "key": {

  17. "algo": "rsa",

  18. "size": 2048

  19. },

  20. "names": [

  21. {

  22. "C": "CN",

  23. "L": "Beijing",

  24. "ST": "Beijing",

  25. "O": "k8s",

  26. "OU": "System"

  27. }

  28. ]

  29. }

  30. EOF

 
   
   
 
  1. cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

  • 2.5.3 制作kube-proxy证书

 
   
   
 
  1. cat << EOF | tee kube-proxy-csr.json

  2. {

  3. "CN": "system:kube-proxy",

  4. "hosts": [],

  5. "key": {

  6. "algo": "rsa",

  7. "size": 2048

  8. },

  9. "names": [

  10. {

  11. "C": "CN",

  12. "L": "Beijing",

  13. "ST": "Beijing",

  14. "O": "k8s",

  15. "OU": "System"

  16. }

  17. ]

  18. }

  19. EOF

 
   
   
 
  1. cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

一个全新的基于容器技术的分布式架构方案-Kubernetes

2.6 安装k8s服务
2.6.1 解压缩文件
 
   
   
 
  1. tar -zxvf kubernetes-server-linux-amd64.tar.gz

  2. cd kubernetes/server/bin/

  3. cp kube-scheduler kube-apiserver kube-controller-manager kubectl /k8s/kubernetes/bin/

2.6.2 安装kube-apiserver
  • 2.6.2.1 部署kube-apiserver组件 创建TLS Bootstrapping Token

 
   
   
 
  1. [root@elasticsearch01 bin]# head -c 16 /dev/urandom | od -An -t x | tr -d ' '

  2. 663eb46fb81c4cf2a9bedb84bea03582


  3. vim /k8s/kubernetes/cfg/token.csv

  4. 663eb46fb81c4cf2a9bedb84bea03582,kubelet-bootstrap,10001,"system:kubelet-bootstrap"

  • 2.6.2.2 创建Apiserver配置文件

 
   
   
 
  1. [root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-apiserver


  2. KUBE_APISERVER_OPTS="--logtostderr=true

  3. --v=4

  4. --etcd-servers=https://192.168.4.71:2379

  5. --bind-address=192.168.4.71

  6. --secure-port=6443

  7. --advertise-address=192.168.4.71

  8. --allow-privileged=true

  9. --service-cluster-ip-range=10.254.0.0/16

  10. --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction

  11. --authorization-mode=RBAC,Node

  12. --enable-bootstrap-token-auth

  13. --token-auth-file=/k8s/kubernetes/cfg/token.csv

  14. --service-node-port-range=30000-50000

  15. --tls-cert-file=/k8s/kubernetes/ssl/server.pem

  16. --tls-private-key-file=/k8s/kubernetes/ssl/server-key.pem

  17. --client-ca-file=/k8s/kubernetes/ssl/ca.pem

  18. --service-account-key-file=/k8s/kubernetes/ssl/ca-key.pem

  19. --etcd-cafile=/k8s/etcd/ssl/ca.pem

  20. --etcd-certfile=/k8s/etcd/ssl/server.pem

  21. --etcd-keyfile=/k8s/etcd/ssl/server-key.pem"

  • 2.6.2.3 创建apiserver systemd文件

 
   
   
 
  1. [root@t71 cfg]# vim /usr/lib/systemd/system/kube-apiserver.service


  2. [Unit]

  3. Description=Kubernetes API Server

  4. Documentation=https://github.com/kubernetes/kubernetes


  5. [Service]

  6. EnvironmentFile=-/k8s/kubernetes/cfg/kube-apiserver

  7. ExecStart=/k8s/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS

  8. Restart=on-failure


  9. [Install]

  10. WantedBy=multi-user.target

  • 2.6.2.4 启动服务

 
   
   
 
  1. systemctl daemon-reload

  2. systemctl enable kube-apiserver

  3. systemctl start kube-apiserver

2.6.3 安装kube-scheduler
  • 2.6.3.1 创建kube-scheduler配置文件

 
   
   
 
  1. [root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-scheduler

  2. KUBE_SCHEDULER_OPTS="--logtostderr=true --v=4 --master=127.0.0.1:8080

  • 2.6.3.2 创建kube-scheduler systemd文件

 
   
   
 
  1. [root@t71 cfg]# vim /usr/lib/systemd/system/kube-scheduler.service


  2. [Unit]

  3. Description=Kubernetes Scheduler

  4. Documentation=https://github.com/kubernetes/kubernetes


  5. [Service]

  6. EnvironmentFile=-/k8s/kubernetes/cfg/kube-scheduler

  7. ExecStart=/k8s/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS

  8. Restart=on-failure


  9. [Install]

  10. WantedBy=multi-user.target

  • 2.6.3.3 启动服务

 
   
   
 
  1. systemctl daemon-reload

  2. systemctl start kube-scheduler

  3. systemctl enable kube-scheduler

2.6.4 安装kube-controller-manager
  • 2.6.3.1 创建kube-controller-managerr配置文件

 
   
   
 
  1. [root@t71 cfg]# vim /k8s/kubernetes/cfg/kube-controller-manager


  2. KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true

  3. --v=4

  4. --master=127.0.0.1:8080

  5. --leader-elect=true

  6. --address=127.0.0.1

  7. --service-cluster-ip-range=10.254.0.0/16

  8. --cluster-name=kubernetes

  9. --cluster-signing-cert-file=/k8s/kubernetes/ssl/ca.pem

  10. --cluster-signing-key-file=/k8s/kubernetes/ssl/ca-key.pem

  11. --root-ca-file=/k8s/kubernetes/ssl/ca.pem

  12. --service-account-private-key-file=/k8s/kubernetes/ssl/ca-key.pem"

  • 2.6.3.2 创建kube-controller-manager systemd文件

 
   
   
 
  1. [root@t71 cfg]# vim /usr/lib/systemd/system/kube-controller-manager.service


  2. [Unit]

  3. Description=Kubernetes Controller Manager

  4. Documentation=https://github.com/kubernetes/kubernetes


  5. [Service]

  6. EnvironmentFile=-/k8s/kubernetes/cfg/kube-controller-manager

  7. ExecStart=/k8s/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS

  8. Restart=on-failure


  9. [Install]

  10. WantedBy=multi-user.target

  • 2.6.3.3 启动服务

 
   
   
 
  1. systemctl daemon-reload

  2. systemctl start kube-controller-manager

  3. systemctl enable kube-controller-manager

2.7 验证kube server
  • 2.7.1 添加环境变量

 
   
   
 
  1. vim /etc/profile

  2. export PATH=/k8s/kubernetes/bin:$PATH

  3. source /etc/profile

  • 2.7.2 get

一个全新的基于容器技术的分布式架构方案-Kubernetes

三、k8s-slave部署

3.1 docker 环境安装
 
   
   
 
  1. yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

  2. yum list docker-ce --showduplicates | sort -r

  3. yum install docker-ce -y

  4. systemctl start docker && systemctl enable docker

3.2 部署kubelet插件

kublet 运行在每个 worker 节点上,接收 kube-apiserver 发送的请求,管理 Pod 容器,执行交互式命令,如exec、run、logs 等; kublet 启动时自动向 kube-apiserver 注册节点信息,内置的 cadvisor 统计和监控节点的资源使用情况; 为确保安全,只开启接收 https 请求的安全端口,对请求进行认证和授权,拒绝未授权的访问(如apiserver、heapster)

  • 3.2.1 安装二进制文件

 
   
   
 
  1. wget https://dl.k8s.io/v1.13.1/kubernetes-node-linux-amd64.tar.gz

  2. tar zxvf kubernetes-node-linux-amd64.tar.gz

  3. cd kubernetes/node/bin/

  4. cp kube-proxy kubelet kubectl /k8s/kubernetes/bin/

  • 3.2.2 复制相关证书到node节点

 
   
   
 
  1. scp *.pem 10.2.8.65:$PWD

  • 3.2.3 创建kubelet bootstrap kubeconfig文件 通过脚本实现

 
   
   
 
  1. [root@t72 cfg]# vim environment.sh


  2. #!/bin/bash

  3. #创建kubelet bootstrapping kubeconfig

  4. BOOTSTRAP_TOKEN=663eb46fb81c4cf2a9bedb84bea03582

  5. KUBE_APISERVER="https://192.168.4.71:6443"

  6. #设置集群参数

  7. kubectl config set-cluster kubernetes

  8. --certificate-authority=/k8s/kubernetes/ssl/ca.pem

  9. --embed-certs=true

  10. --server=${KUBE_APISERVER}

  11. --kubeconfig=bootstrap.kubeconfig


  12. #设置客户端认证参数

  13. kubectl config set-credentials kubelet-bootstrap

  14. --token=${BOOTSTRAP_TOKEN}

  15. --kubeconfig=bootstrap.kubeconfig


  16. # 设置上下文参数

  17. kubectl config set-context default

  18. --cluster=kubernetes

  19. --user=kubelet-bootstrap

  20. --kubeconfig=bootstrap.kubeconfig


  21. # 设置默认上下文

  22. kubectl config use-context default --kubeconfig=bootstrap.kubeconfig


  23. #----------------------


  24. # 创建kube-proxy kubeconfig文件


  25. kubectl config set-cluster kubernetes

  26. --certificate-authority=/k8s/kubernetes/ssl/ca.pem

  27. --embed-certs=true

  28. --server=${KUBE_APISERVER}

  29. --kubeconfig=kube-proxy.kubeconfig


  30. kubectl config set-credentials kube-proxy

  31. --client-certificate=/k8s/kubernetes/ssl/kube-proxy.pem

  32. --client-key=/k8s/kubernetes/ssl/kube-proxy-key.pem

  33. --embed-certs=true

  34. --kubeconfig=kube-proxy.kubeconfig


  35. kubectl config set-context default

  36. --cluster=kubernetes

  37. --user=kube-proxy

  38. --kubeconfig=kube-proxy.kubeconfig


  39. kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

  • 3.2.4 执行脚本

 
   
   
 
  1. sh environment.sh

  • 3.2.5 创建kubelet参数配置模板文件

 
   
   
 
  1. [root@t72 cfg]# vim kubelet.config


  2. kind: KubeletConfiguration

  3. apiVersion: kubelet.config.k8s.io/v1beta1

  4. address: 192.168.4.72

  5. port: 10250

  6. readOnlyPort: 10255

  7. cgroupDriver: cgroupfs

  8. clusterDNS: ["10.254.0.10"]

  9. clusterDomain: cluster.local.

  10. failSwapOn: false

  11. #authentication:

  12. # anonymous:

  13. # enabled: true

  14. authentication:

  15. anonymous:

  16. enabled: true # Defaults to false as of 1.10

  17. webhook:

  18. enabled: false # Deafults to true as of 1.10

  19. authorization:

  20. mode: AlwaysAllow

  • 3.2.6 创建kubelet配置文件

 
   
   
 
  1. [root@t72 cfg]# vim kubelet


  2. KUBELET_OPTS="--logtostderr=true

  3. --v=4

  4. --hostname-override=192.168.4.72

  5. --kubeconfig=/k8s/kubernetes/cfg/kubelet.kubeconfig

  6. --bootstrap-kubeconfig=/k8s/kubernetes/cfg/bootstrap.kubeconfig

  7. --config=/k8s/kubernetes/cfg/kubelet.config

  8. --cert-dir=/k8s/kubernetes/ssl

  9. --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"

  • 3.2.7 创建kubelet systemd文件

 
   
   
 
  1. [root@t72 cfg]# vim /usr/lib/systemd/system/kubelet.service


  2. [Unit]

  3. Description=Kubernetes Kubelet

  4. After=docker.service

  5. Requires=docker.service


  6. [Service]

  7. EnvironmentFile=-/k8s/kubernetes/cfg/kubelet

  8. ExecStart=/k8s/kubernetes/bin/kubelet $KUBELET_OPTS

  9. Restart=on-failure

  10. KillMode=process


  11. [Install]

  12. WantedBy=multi-user.target

  • 3.2.8 在master节点上面操作,将kubelet-bootstrap用户绑定到系统集群角色

 
   
   
 
  1. kubectl create clusterrolebinding kubelet-bootstrap

  2. --clusterrole=system:node-bootstrapper

  3. --user=kubelet-bootstrap

  • 3.2.9 启动kubelet服务

 
   
   
 
  1. systemctl daemon-reload

  2. systemctl start kubelet

  3. systemctl enable kubelet

  • 3.2.10 Master接受kubelet CSR请求 可以手动或自动 approve CSR 请求。推荐使用自动的方式,因为从 v1.8 版本开始,可以自动轮转approve csr 后生成的证书,如下是手动 approve CSR请求操作方法 查看CSR列表

 
   
   
 
  1. # kubectl get csr

  2. NAME AGE REQUESTOR CONDITION

  3. node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc 102s kubelet-bootstrap Pending

接受node

 
   
   
 
  1. # kubectl certificate approve node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc

  2. certificatesigningrequest.certificates.k8s.io/node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc approved

再看CRS

 
   
   
 
  1. kubectl get csr

  2. NAME AGE REQUESTOR CONDITION

  3. node-csr-ij3py9j-yi-eoa8sOHMDs7VeTQtMv0N3Efj3ByZLMdc 5m13s kubelet-bootstrap Approved,Issued

3.3 安装kube-proxy

  • 3.3.1 kube-proxy的配置文件

 
   
   
 
  1. [root@t72 cfg]# vim kube-proxy


  2. KUBE_PROXY_OPTS="--logtostderr=true

  3. --v=4

  4. --hostname-override=192.168.4.72

  5. --cluster-cidr=10.254.0.0/16

  6. --kubeconfig=/k8s/kubernetes/cfg/kube-proxy.kubeconfig"

  • 3.3.2 kube-proxy的systemd 文件

 
   
   
 
  1. [root@t72 cfg]# vim /usr/lib/systemd/system/kube-proxy.service


  2. [Unit]

  3. Description=Kubernetes Proxy

  4. After=network.target


  5. [Service]

  6. EnvironmentFile=-/k8s/kubernetes/cfg/kube-proxy

  7. ExecStart=/k8s/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS

  8. Restart=on-failure


  9. [Install]

  10. WantedBy=multi-user.target

  • 3.3.3 启动

 
   
   
 
  1. systemctl daemon-reload

  2. systemctl start kube-proxy

  3. systemctl enable kube-proxy

3.4 部署另外一个node和认证csr

注意: 安装期间,如果kubelet,kube-proxy配置错误,比如监听IP或者hostname错误导致node not found,需要删除kubelet-client证书,重启kubelet服务,重启认证csr即可

四、 网络部署

4.1 etcd注册网段(master节点,因为只有master安装了etcd)

 
   
   
 
  1. /k8s/etcd/bin/etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://192.168.4.71:2379" set /k8s/network/config '{ "Network": "10.254.0.0/16", "Backend": {"Type": "vxlan"}}'

4.2 Flanneld部署

  • 4.2.1 解压

 
   
   
 
  1. tar -xvf flannel-v0.10.0-linux-amd64.tar.gz

  2. mv flanneld mk-docker-opts.sh /k8s/kubernetes/bin/

  • 4.2.2 配置flanneld

 
   
   
 
  1. [root@t72 cfg]# vim flanneld


  2. FLANNEL_OPTIONS="--etcd-endpoints=https://192.168.4.71:2379, -etcd-cafile=/k8s/etcd/ssl/ca.pem -etcd-certfile=/k8s/etcd/ssl/server.pem -etcd-keyfile=/k8s/etcd/ssl/server-key.pem -etcd-prefix=/k8s/network"

  • 4.2.3 创建flanneld systemd文件

 
   
   
 
  1. [root@t72 cfg]# vim /usr/lib/systemd/system/flanneld.service


  2. [Unit]

  3. Description=Flanneld overlay address etcd agent

  4. After=network-online.target network.target

  5. Before=docker.service


  6. [Service]

  7. Type=notify

  8. EnvironmentFile=/k8s/kubernetes/cfg/flanneld

  9. ExecStart=/k8s/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS

  10. #ExecStart=/k8s/kubernetes/bin/flanneld $FLANNEL_OPTIONS

  11. ExecStartPost=/k8s/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env

  12. Restart=on-failure


  13. [Install]

  14. WantedBy=multi-user.target

注: mk-docker-opts.sh 脚本将分配给 flanneld 的 Pod 子网网段信息写入 /run/flannel/docker 文件,后续 docker 启动时 使用这个文件中的环境变量配置 docker0 网桥;flanneld 使用系统缺省路由所在的接口与其它节点通信,对于有多个网络接口(如内网和公网)的节点,可以用 -iface 参数指定通信接口; flanneld 运行时需要 root 权限;

  • 4.2.4 配置Docker启动指定子网

修改EnvironmentFile=/run/flannel/subnet.env,ExecStart=/usr/bin/dockerd $DOCKERNETWORKOPTIONS即可

 
   
   
 
  1. [root@t72 cfg]# cat /usr/lib/systemd/system/docker.service | grep -v "#"

  2. [Unit]

  3. Description=Docker Application Container Engine

  4. Documentation=https://docs.docker.com

  5. BindsTo=containerd.service

  6. After=network-online.target firewalld.service containerd.service

  7. Wants=network-online.target

  8. Requires=docker.socket


  9. [Service]

  10. Type=notify

  11. EnvironmentFile=/run/flannel/subnet.env

  12. ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS

  13. ExecReload=/bin/kill -s HUP $MAINPID

  14. TimeoutSec=0

  15. RestartSec=2

  16. Restart=always


  17. StartLimitBurst=3


  18. StartLimitInterval=60s


  19. LimitNOFILE=infinity

  20. LimitNPROC=infinity

  21. LimitCORE=infinity


  22. TasksMax=infinity


  23. Delegate=yes


  24. KillMode=process


  25. [Install]

  26. WantedBy=multi-user.target

  • 4.2.5 启动服务

注: 启动flannel前要关闭docker及相关的kubelet这样flannel才会覆盖docker0网桥(如果已经存在docker0,则先删除docker0 ip link delete docker0)

 
   
   
 
  1. systemctl daemon-reload

  2. systemctl stop docker

  3. systemctl start flanneld

  4. systemctl enable flanneld

  5. systemctl start docker

  6. systemctl restart kubelet

  7. systemctl restart kube-proxy

  • 4.2.6 验证服务

 
   
   
 
  1. ip addr

一个全新的基于容器技术的分布式架构方案-Kubernetes

 
   
   
 
  1. kubectl get nodes -o wide

一个全新的基于容器技术的分布式架构方案-Kubernetes

五、后记

5.1 kubernetes 组件图


Master节点:Master节点上面主要由四个模块组成,APIServer,schedule,controller-manager,etcd

  • APIServer: APIServer负责对外提供RESTful的kubernetes API的服务,它是系统管理指令的统一接口,任何对资源的增删改查都要交给APIServer处理后再交给etcd,如图,kubectl(kubernetes提供的客户端工具,该工具内部是对kubernetes API的调用)是直接和APIServer交互的。

  • schedule: schedule负责调度Pod到合适的Node上,如果把scheduler看成一个黑匣子,那么它的输入是pod和由多个Node组成的列表,输出是Pod和一个Node的绑定。kubernetes目前提供了调度算法,同样也保留了接口。用户根据自己的需求定义自己的调度算法。

  • controller manager: 如果APIServer做的是前台的工作的话,那么controller manager就是负责后台的。每一个资源都对应一个控制器。而control manager就是负责管理这些控制器的,比如我们通过APIServer创建了一个Pod,当这个Pod创建成功后,APIServer的任务就算完成了。

  • etcd:etcd是一个高可用的键值存储系统,kubernetes使用它来存储各个资源的状态,从而实现了Restful的API。

Node节点:每个Node节点主要由三个模板组成:kublet, kube-proxy

  • kube-proxy: 该模块实现了kubernetes中的服务发现和反向代理功能。kube-proxy支持TCP和UDP连接转发,默认基Round Robin算法将客户端流量转发到与service对应的一组后端pod。服务发现方面,kube-proxy使用etcd的watch机制监控集群中service和endpoint对象数据的动态变化,并且维护一个service到endpoint的映射关系,从而保证了后端pod的IP变化不会对访问者造成影响,另外,kube-proxy还支持session affinity。

  • kublet:kublet是Master在每个Node节点上面的agent,是Node节点上面最重要的模块,它负责维护和管理该Node上的所有容器,但是如果容器不是通过kubernetes创建的,它并不会管理。本质上,它负责使Pod的运行状态与期望的状态一致。

5.2 Kubernetes工作流程

一个全新的基于容器技术的分布式架构方案-Kubernetes以创建Deployment为例:

  1. Devops人员通过Apiserver,访问k8s,发送创建Deployment请求

  2. Apiserver将Deployment元数据存至etcd中

  3. Controller manager 中的Deployment controller通过Apiserver watch 资源的变换情况,发现有新的Deployment需要创建

  4. Deployment controller请求Apiserver创建ReplicaSet资源, Apiserver将ReplicaSet元数据存至etcd中

  5. Controller manager 中的ReplicaSet controller通过Apiserver watch 资源的变换情况,发现有新的ReplicaSet需要创建

  6. ReplicaSet controller请求Apiserver创建Pod资源, Apiserver将Pod元数据存至etcd中

  7. Scheduler通过Apiserver watch 资源的变换情况,发现有新的Pod需要创建。综合节点资源通过(请求apiserver获取)和pod资源情况(请求apiserver获取)制定出调度信息返回给给Apiserver

  8. Apiserver 根据Scheduler返回的调度信息,通过slave中kubelet组件创建pod资源

  9. kubelet通过docker daemon 创建container资源(包括用户容器和pause容器)

5.3 kubernetes 网络架构图(flanneld)

一个全新的基于容器技术的分布式架构方案-Kubernetes

  • 数据从源容器中发出后,经由所在主机的docker0虚拟网卡转发到flannel0虚拟网卡,这是个P2P的虚拟网卡,flanneld服务监听在网卡的另外一端。

  • Flannel通过Etcd服务维护了一张节点间的路由表,在稍后的配置部分我们会介绍其中的内容。

  • 源主机的flanneld服务将原本的数据内容UDP封装后根据自己的路由表投递给目的节点的flanneld服务,数据到达以后被解包,然后直接进入目的节点的flannel0虚拟网卡, 然后被转发到目的主机的docker0虚拟网卡,最后就像本机容器通信一下的有docker0路由到达目标容器。

5.4 kubernetes应用模式


以上是关于一个全新的基于容器技术的分布式架构方案-Kubernetes的主要内容,如果未能解决你的问题,请参考以下文章

Kubernetes

k8s

Kubernetes是啥?

k8s

centos7基于kubeadm安装部署Kubernetes (1.15.2)集群

新一代分布式架构Kubernetes(K8S)入门知识点总结