Etcd 集群搭建

Posted 2021-08-27 X糊涂仙儿

1、集群

主机名	ip	配置
etcd-01	192.168.1.61	8C/16G/100G
etcd-02	192.168.1.62	8C/16G/100G
etcd-03	192.168.1.63	8C/16G/100G

2、配置etcd集群

1> hosts文件

[root@k01 ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.61 k01
192.168.1.62 k02
192.168.1.63 k03
192.168.1.71 node01
192.168.1.72 node02

2> 创建 ca证书

mkdir -pv etcdssl
cd etcdssl

创建ca-config、ca-csr 

cat ca-config.json << EOF
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "www": {
         "expiry": "87600h",
         "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ]
      }
    }
  }
}
EOF

cat ca-csr.json << EOF
{
    "CN": "etcd CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "Beijing",
            "ST": "Beijing"
        }
    ]
}
EOF

生成证书

[root@k01 etcdssl]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
2021/08/20 21:05:48 [INFO] generating a new CA key and certificate from CSR
2021/08/20 21:05:48 [INFO] generate received request
2021/08/20 21:05:48 [INFO] received CSR
2021/08/20 21:05:48 [INFO] generating key: rsa-2048
2021/08/20 21:05:48 [INFO] encoded CSR
2021/08/20 21:05:48 [INFO] signed certificate with serial number 437xxxxxxxxxxxxxx394879734494661358352

查看

[root@k01 etcdssl]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

3> 使用ca签发etcd证书

创建etcd证书签名请求文件

[root@k01 etcdssl]# cat server-csr.json 
{
    "CN": "etcd",
    "hosts": [
    "192.168.1.61",
    "192.168.1.62",
    "192.168.1.63",
    "192.168.1.64"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "BeiJing",
            "ST": "BeiJing"
        }
    ]
}

生成证书

[root@k01 etcdssl]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server
2021/08/20 21:07:05 [INFO] generate received request
2021/08/20 21:07:05 [INFO] received CSR
2021/08/20 21:07:05 [INFO] generating key: rsa-2048
2021/08/20 21:07:06 [INFO] encoded CSR
2021/08/20 21:07:06 [INFO] signed certificate with serial number 5579849624423xxxxxxxxxxxxxxxxx91627608515054481
2021/08/20 21:07:06 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

证书分发至其它节点

创建目录

[root@k01 etcdssl]# ansible etcd -m shell -a "mkdir -p /opt/etcd/ssl"

创建分发脚本

vim scpetcdssl.sh
#！/bin/bash

for node in k01 k02 k03 
do
	ssh root@${node} "mkdir -pv /opt/etcd/ssl"
	scp ./ca*.pem  root@${node}:/opt/etcd/ssl
	scp ./server*.pem  root@${node}:/opt/etcd/ssl
done

三、部署Etcd

解压创建配置文件

tar xzvf etcd-v3.4.16-linux-amd64.tar.gz
cp -a etcd-v3.4.16-linux-amd64/etcd* /usr/local/bin/
ansible etcd -m shell -a "mkdir -pv /opt/etcd/cfg"

cat >/opt/etcd/cfg/etcd.conf <<EOF
#[Member]
ETCD_NAME="etcd-01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.61:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.61:2379,http://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.61:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.61:2379"
ETCD_INITIAL_CLUSTER="etcd-01=https://192.168.1.61:2380,etcd-02=https://192.168.1.62:2380,etcd-03=https://192.168.1.63:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

[root@k01 ~]# cat /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/usr/local/bin/etcd \\
--cert-file=/opt/etcd/ssl/server.pem \\
--key-file=/opt/etcd/ssl/server-key.pem \\
--peer-cert-file=/opt/etcd/ssl/server.pem \\
--peer-key-file=/opt/etcd/ssl/server-key.pem \\
--trusted-ca-file=/opt/etcd/ssl/ca.pem \\
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \\
--logger=zap
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF

注：
ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群

启动服务

systemctl start etcd.service
systemctl enable etcd.service
[root@k01 ~]# systemctl status etcd.service
● etcd.service - Etcd Server
   Loaded: loaded (/usr/lib/systemd/system/etcd.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2021-08-22 18:26:28 CST; 5h 11min ago
 Main PID: 17312 (etcd)
    Tasks: 15
   Memory: 54.3M
   CGroup: /system.slice/etcd.service
           └─17312 /usr/local/bin/etcd --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --tr...

Aug 22 23:36:04 k01 etcd[17312]: {"level":"warn","ts":"2021-08-22T23:36:04.194+0800","caller":"rafthttp/probing_status.go:86","msg":"prober found high clock drift","round-tripper-name":"ROUND_...":"30.14026ms"}

健康状态检查

[root@k01 ~]# etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.1.61:2379,https://192.168.1.62:2379,https://192.168.1.63:2379" endpoint health
https://192.168.1.61:2379 is healthy: successfully committed proposal: took = 57.708307ms
https://192.168.1.62:2379 is healthy: successfully committed proposal: took = 59.868422ms
https://192.168.1.63:2379 is healthy: successfully committed proposal: took = 94.302948ms