linux运维架构之路-Kubernetes1.13离线集群部署双向认证

Posted yanxinjiang

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了linux运维架构之路-Kubernetes1.13离线集群部署双向认证相关的知识,希望对你有一定的参考价值。

一、部署环境

1、服务器规划

IP地址

主机名

CPU

内存

10.0.0.100

k8s-master01

2C

2G

10.0.0.101

k8s-node01

2C

2G

10.0.0.102

k8s-node02

2C

2G

2、Kubernetes 1.13包下载

https://github.com/kubernetes/kubernetes/releases

所用到包的版本:

技术图片

3、Kubernetes工作流程

技术图片

4、Kubernetes部署前准备

①系统环境

[root@k8s-master01 ~]# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
[root@k8s-master01 ~]# uname -r
3.10.0-957.21.3.el7.x86_64
[root@k8s-master01 ~]# systemctl status firewalld.service 
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)
[root@k8s-master01 ~]# getenforce
Disabled

②关闭Swap

swapoff -a && sysctl -w vm.swappiness=0
vim /etc/fstab
UUID=b6ff8dda-f2db-49ad-a091-bdd4e82cdace /                       xfs     defaults        0 0
UUID=f4eaa64a-f9c7-4e11-a78b-13d423ac65d2 /boot                   xfs     defaults        0 0
#UUID=bc4d148a-56e0-4fa8-96a2-d10e5470963c swap                    swap    defaults        0 0

③设置Docker所需参数

cat << EOF | tee /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl -p /etc/sysctl.d/k8s.conf

 二、部署 Docker(所有节点)

1、安装

yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
yum list docker-ce --showduplicates | sort -r
yum install docker-ce -y
systemctl start docker && systemctl enable docker

2、创建安装所需目录

mkdir /k8s/etcd/bin,cfg,ssl -p
mkdir /k8s/kubernetes/bin,cfg,ssl -p

3、安装生成证书工具CFSSL(Master节点)

mkdir tools && cd tools
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

4、创建认证证书(Master节点)

①创建 ETCD 证书

cat << EOF | tee ca-config.json

  "signing": 
    "default": 
      "expiry": "87600h"
    ,
    "profiles": 
      "www": 
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      
    
  

EOF

创建 ETCD CA 配置文件  

cat << EOF | tee ca-csr.json

    "CN": "etcd CA",
    "key": 
        "algo": "rsa",
        "size": 2048
    ,
    "names": [
        
            "C": "CN",
            "L": "Shenzhen",
            "ST": "Shenzhen"
        
    ]

EOF

创建 ETCD Server 证书

cat << EOF | tee server-csr.json

    "CN": "etcd",
    "hosts": [
    "10.0.0.100",
    "10.0.0.101",
    "10.0.0.102"
    ],
    "key": 
        "algo": "rsa",
        "size": 2048
    ,
    "names": [
        
            "C": "CN",
            "L": "Shenzhen",
            "ST": "Shenzhen"
        
    ]

EOF

生成 ETCD CA 证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

②创建 Kubernetes CA 证书

cat << EOF | tee ca-config.json

  "signing": 
    "default": 
      "expiry": "87600h"
    ,
    "profiles": 
      "kubernetes": 
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      
    
  

EOF
cat << EOF | tee ca-csr.json

    "CN": "kubernetes",
    "key": 
        "algo": "rsa",
        "size": 2048
    ,
    "names": [
        
            "C": "CN",
            "L": "Shenzhen",
            "ST": "Shenzhen",
            "O": "k8s",
            "OU": "System"
        
    ]

EOF

生成Kubernetes 证书和秘钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

③生成API_SERVER证书

cat << EOF | tee server-csr.json

    "CN": "kubernetes",
    "hosts": [
      "10.0.0.1",
      "127.0.0.1",
      "10.0.0.100",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": 
        "algo": "rsa",
        "size": 2048
    ,
    "names": [
        
            "C": "CN",
            "L": "Shenzhen",
            "ST": "Shenzhen",
            "O": "k8s",
            "OU": "System"
        
    ]

EOF

生成API_SERVER证书和秘钥

 

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

 

④创建 Kubernetes Proxy 证书

cat << EOF | tee kube-proxy-csr.json

  "CN": "system:kube-proxy",
  "hosts": [],
  "key": 
    "algo": "rsa",
    "size": 2048
  ,
  "names": [
    
      "C": "CN",
      "L": "Shenzhen",
      "ST": "Shenzhen",
      "O": "k8s",
      "OU": "System"
    
  ]

EOF

生成Kubernetes Proxy 证书和秘钥

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

5、ssh-key免秘钥认证

ssh-keygen

ssh-copy-id 10.0.0.101
ssh-copy-id 10.0.0.102

三、部署ETCD服务

1、上传解压文件

tar xf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64/
cp etcd etcdctl /k8s/etcd/bin/

2、创建ETCD配置文件

vim /k8s/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.100:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.100:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.100:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.100:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.0.0.100:2380,etcd02=https://10.0.0.101:2380,etcd03=https://10.0.0.102:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

创建 etcd的 systemd unit 文件

vim /usr/lib/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/k8s/etcd/cfg/etcd
ExecStart=/k8s/etcd/bin/etcd --name=$ETCD_NAME --data-dir=$ETCD_DATA_DIR --listen-peer-urls=$ETCD_LISTEN_PEER_URLS --listen-client-urls=$ETCD_LISTEN_CLIENT_URLS,http://127.0.0.1:2379 \\
--advertise-client-urls=$ETCD_ADVERTISE_CLIENT_URLS --initial-advertise-peer-urls=$ETCD_INITIAL_ADVERTISE_PEER_URLS --initial-cluster=$ETCD_INITIAL_CLUSTER --initial-cluster-token=$ETCD_INITIAL_CLUSTER_TOKEN --initial-cluster-state=new --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --peer-cert-file=/k8s/etcd/ssl/server.pem --peer-key-file=/k8s/etcd/ssl/server-key.pem --trusted-ca-file=/k8s/etcd/ssl/ca.pem --peer-trusted-ca-file=/k8s/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

拷贝证书文件

cd tools
cp ca*pem server*pem /k8s/etcd/ssl

将启动文件、配置文件分发到 节点1、节点2

cd /k8s
scp -r etcd 10.0.0.101:/k8s/
scp -r etcd 10.0.0.102:/k8s/
scp /usr/lib/systemd/system/etcd.service  10.0.0.101:/usr/lib/systemd/system/etcd.service
scp /usr/lib/systemd/system/etcd.service  10.0.0.102:/usr/lib/systemd/system/etcd.service

修改节点1、节点2配置文件

vim /k8s/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.101:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.101:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.101:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.101:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.0.0.100:2380,etcd02=https://10.0.0.101:2380,etcd03=https://10.0.0.102:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
vim /k8s/etcd/cfg/etcd

#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://10.0.0.102:2380"
ETCD_LISTEN_CLIENT_URLS="https://10.0.0.102:2379"

#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.0.0.102:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://10.0.0.102:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://10.0.0.100:2380,etcd02=https://10.0.0.101:2380,etcd03=https://10.0.0.102:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

3、启动ETCD服务

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

4、验证集群是否正常运行

/k8s/etcd/bin/etcdctl --ca-file=/k8s/etcd/ssl/ca.pem --cert-file=/k8s/etcd/ssl/server.pem --key-file=/k8s/etcd/ssl/server-key.pem --endpoints="https://10.0.0.100:2379,\\
https://10.0.0.101:2379,\\
https://10.0.0.102:2379" cluster-health
member 5db3ea816863435 is healthy: got healthy result from https://10.0.0.102:2379
member 991b5845cecb31b is healthy: got healthy result from https://10.0.0.101:2379
member c67ee2780d64a0d4 is healthy: got healthy result from https://10.0.0.100:2379
cluster is healthy

注意:
启动ETCD集群同时启动二个节点,启动一个节点集群是无法正常启动的;

 

以上是关于linux运维架构之路-Kubernetes1.13离线集群部署双向认证的主要内容,如果未能解决你的问题,请参考以下文章

linux运维架构之路-linux基础优化

linux运维架构之路-Nginx提高

linux运维架构之路-redis集群

linux运维架构之路-Oracle基本操作

linux运维架构之路-shell编程入门

linux运维架构之路-SaltStack快速入门