三二进制高可用

Posted 2021-12-31 哭泣的馒头

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了三二进制高可用相关的知识，希望对你有一定的参考价值。

一、环境介绍

使用一中部署的基础环境

192.168.10.131：作为master节点和证书颁发服务器
192.168.10.132：作为master节点，node节点，打污点
192.168.10.133：作为master节点，node节点
192.168.10.188：作为apiserver集群虚拟节点
10.255.0.0/16 service地址
10.0.0.0/16 pod地址

二、集群证书

1、安装签发证书工具cfssl

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

2、生成ca证书json文件，创建证书

mkdir -p /data/work && cd /data/work/
cat>ca-config.json <<EOF

  "signing": 
      "default": 
          "expiry": "87600h"
        ,
      "profiles": 
          "kubernetes": 
              "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
              ],
              "expiry": "87600h"
          
      
  

EOF

cat> ca-csr.json <<EOF

  "CN": "kubernetes",
  "key": 
      "algo": "rsa",
      "size": 2048
  ,
  "names": [
    
      "C": "CN",
      "ST": "shandong",
      "L": "jinan",
      "O": "k8s",
      "OU": "system"
    
  ],
  "ca": 
          "expiry": "87600h"
  

EOF

cfssl gencert -initca ca-csr.json  | cfssljson -bare ca

三、搭建ETCD

1、创建etcd证书

cat> etcd-csr.json <<EOF

  "CN": "etcd",
  "hosts": [
    "127.0.0.1",
    "192.168.10.131",
    "192.168.10.132",
    "192.168.10.133",
    "192.168.10.134"
  ],
  "key": 
    "algo": "rsa",
    "size": 2048
  ,
  "names": [
    "C": "CN",
    "ST": "shandong",
    "L": "jinan",
    "O": "k8s",
    "OU": "system"
  ]

EOF
#上述文件hosts字段中IP为所有etcd节点的集群内部通信IP，可以预留几个，做扩容用。

2、生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
ls etcd*

3、部署etcd集群

把etcd-v3.4.13-linux-amd64.tar.gz上传到/data/work目录下
tar -xf etcd-v3.4.13-linux-amd64.tar.gz
scp -r etcd-v3.4.13-linux-amd64/etcd* k8s01:/usr/local/bin/
scp -r etcd-v3.4.13-linux-amd64/etcd* k8s02:/usr/local/bin/
scp -r etcd-v3.4.13-linux-amd64/etcd* k8s03:/usr/local/bin/

创建配置文件和启动文件
cat> etcd.conf <<EOF
#[Member]
ETCD_NAME="etcd-node1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://etcd-now:2380"
ETCD_LISTEN_CLIENT_URLS="https://etcd-now:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd-now:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://etcd-now:2379"
ETCD_INITIAL_CLUSTER="cluster-ip"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
cat >etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
 
[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \\
  --cert-file=/etc/etcd/ssl/etcd.pem \\
  --key-file=/etc/etcd/ssl/etcd-key.pem \\
  --trusted-ca-file=/etc/etcd/ssl/ca.pem \\
  --peer-cert-file=/etc/etcd/ssl/etcd.pem \\
  --peer-key-file=/etc/etcd/ssl/etcd-key.pem \\
  --peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\
  --peer-client-cert-auth \\
  --client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
 
[Install]
WantedBy=multi-user.target
EOF

启动ETCD集群
for ip in k8s01 k8s02 k8s03;do
  ssh root@$ip "mkdir -pv /etc/etcd/ssl;mkdir -pv /var/lib/etcd/default.etcd"
  scp ca*.pem  root@$ip:/etc/etcd/ssl
  scp etcd*.pem  root@$ip:/etc/etcd/ssl
  scp etcd.conf  root@$ip:/etc/etcd/
  scp etcd.service  root@$ip:/usr/lib/systemd/system/
done

k8s01操作：
sed -i s/etcd-node1/etcd-k8s01/g /etc/etcd/etcd.conf
sed -i s/etcd-now/192.168.10.131/g /etc/etcd/etcd.conf
sed -i s#cluster-ip#etcd-k8s01=https://192.168.10.131:2380,etcd-k8s02=https://192.168.10.132:2380,etcd-k8s03=https://192.168.10.133:2380#g /etc/etcd/etcd.conf
systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service
systemctl status etcd

k8s02操作：
sed -i s/etcd-node1/etcd-k8s02/g /etc/etcd/etcd.conf
sed -i s/etcd-now/192.168.10.132/g /etc/etcd/etcd.conf
sed -i s#cluster-ip#etcd-k8s01=https://192.168.10.131:2380,etcd-k8s02=https://192.168.10.132:2380,etcd-k8s03=https://192.168.10.133:2380#g /etc/etcd/etcd.conf
systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service
systemctl status etcd

k8s03操作：
sed -i s/etcd-node1/etcd-k8s03/g /etc/etcd/etcd.conf
sed -i s/etcd-now/192.168.10.133/g /etc/etcd/etcd.conf
sed -i s#cluster-ip#etcd-k8s01=https://192.168.10.131:2380,etcd-k8s02=https://192.168.10.132:2380,etcd-k8s03=https://192.168.10.133:2380#g /etc/etcd/etcd.conf
systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service
systemctl status etcd

启动etcd的时候，先启动第一个节点的etcd服务，会一直卡住在启动的状态，然后接着再启动其他节点的etcd，这样第一个节点etcd才会正常起来
ETCD_NAME和ETCD_INITIAL_CLUSTER中的必须对应起来，如果不一致没法启动

查看集群
ETCDCTL_API=3
etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.10.131:2379,https://192.168.10.132:2379,https://192.168.10.133:2379  endpoint health

四、安装kubernetes组件

1、分发安装包

把kubernetes-server-linux-amd64.tar.gz上传到master的/data/work
tar zxvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/

master节点：
for ip in k8s01 k8s02 k8s03;do
  ssh root@$ip "mkdir -pv /etc/kubernetes/ssl;mkdir -pv /var/log/kubernetes"
  scp kube-apiserver  root@$ip:/usr/local/bin/
  scp kube-controller-manager  root@$ip:/usr/local/bin/
  scp kube-scheduler  root@$ip:/usr/local/bin/
  scp kubectl  root@$ip:/usr/local/bin/
done

node节点：
for ip in k8s02 k8s03;do
  scp kubelet  root@$ip:/usr/local/bin/
  scp kube-proxy  root@$ip:/usr/local/bin/
done

2、部署api-server

#创建token.csv文件
cd /data/work/
cat > token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d  ),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF
cat token.csv
cat >kube-apiserver-csr.json <<EOF

  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.10.131",
    "192.168.10.132",
    "192.168.10.133",
    "192.168.10.188",
    "10.255.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
  ],
  "key": 
    "algo": "rsa",
    "size": 2048
  ,
  "names": [
    
      "C": "CN",
      "ST": "shandong",
      "L": "jinan",
      "O": "k8s",
      "OU": "system"
    
  ]

EOF
host：localhost地址 + master部署节点的ip地址 + etcd节点的部署地址 + 负载均衡指定的VIP(192.168.10.188) + service ip段的第一个合法地址(10.255.0.1) + k8s默认指定的一些地址。
证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
配置文件
cat >kube-apiserver.conf <<EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --anonymous-auth=false \\
  --bind-address=apiserver-node \\
  --secure-port=6443 \\
  --advertise-address=apiserver-node \\
  --insecure-port=0 \\
  --authorization-mode=Node,RBAC \\
  --runtime-config=api/all=true \\
  --enable-bootstrap-token-auth \\
  --service-cluster-ip-range=10.255.0.0/16 \\
  --token-auth-file=/etc/kubernetes/token.csv \\
  --service-node-port-range=30000-50000 \\
  --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \\
  --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \\
  --client-ca-file=/etc/kubernetes/ssl/ca.pem \\
  --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \\
  --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \\
  --service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \\
  --service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \\
  --service-account-issuer=https://kubernetes.default.svc.cluster.local \\
  --etcd-cafile=/etc/etcd/ssl/ca.pem \\
  --etcd-certfile=/etc/etcd/ssl/etcd.pem \\
  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\
  --etcd-servers=https://192.168.10.131:2379,https://192.168.10.132:2379,https://192.168.10.133:2379 \\
  --enable-swagger-ui=true \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
k8s多Master集群二进制部署
 k8s多Master集群二进制部署
 k8s多Master集群二进制部署
 k8s多Master集群二进制部署
 MHA-结合MySQL半同步复制高可用集群(Centos7)
 关于CUDA5之后cutil.h不可用的问题