CentOS 7 ETCD 集群配置大全

Posted k8s技术圈

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了CentOS 7 ETCD 集群配置大全相关的知识,希望对你有一定的参考价值。

Etcd 是 CoreOS 基于 Raft 开发的分布式 key-value 存储,可用于服务发现、共享配置以及一致性保障(如数据库选主、分布式锁等)。Etcd 集群配置分为三种:

  • 静态发现

  • Etcd 动态发现

  • DNS 动态发现:通过 DNS 的 SRV 解析动态发现集群

本文主要基于静态发现 和 DNS 动态发现两种方式,并结合自签的TLS证书来创建集群。

环境准备

安装

在三台机器上均执行:

 
   
   
 
  1. [root@node01 ~]# yum install etcd -y

  2. [root@node01 ~]# rpm -qa etcd

  3. etcd-3.3.11-2.el7.centos.x86_64

创建Etcd所需目录,在三台机器上均执行:

 
   
   
 
  1. mkdir /data/k8s/etcd/{data,wal} -p

  2. mkdir -p /etc/kubernetes/cert

  3. chown -R etcd.etcd /data/k8s/etcd

静态集群

node01 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="http://192.168.1.91:2380"

  4. ETCD_LISTEN_CLIENT_URLS="http://192.168.1.91:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd1"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.91:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.91:2379"


  13. ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"

node02 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="http://192.168.1.92:2380"

  4. ETCD_LISTEN_CLIENT_URLS="http://192.168.1.92:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd2"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.92:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.92:2379"


  13. ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"

node03 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="http://192.168.1.93:2380"

  4. ETCD_LISTEN_CLIENT_URLS="http://192.168.1.93:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd3"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.1.93:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="http://192.168.1.93:2379"


  13. ETCD_INITIAL_CLUSTER="etcd1=http://192.168.1.91:2380,etcd2=http://192.168.1.92:2380,etcd3=http://192.168.1.93:2380"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"

启动测试:

 
   
   
 
  1. [root@node01 etcd]# systemctl start etcd

  2. [root@node01 etcd]# systemctl status etcd

  3. etcd.service - Etcd Server

  4. Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)

  5. Active: active (running) since Thu 2019-11-07 09:28:54 CST; 5s ago

  6. Main PID: 1546 (etcd)

  7. Tasks: 8

  8. Memory: 41.3M

  9. CGroup: /system.slice/etcd.service

  10. └─1546 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=http://192.168.1.91:2379


  11. Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 [term: 1] received a MsgVote message with higher term from 9c64fba479c5e94 [term: 2]

  12. Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 became follower at term 2

  13. Nov 07 09:28:54 node01.k8s.com etcd[1546]: 3b8b38de05e2c497 [logterm: 1, index: 3, vote: 0] cast MsgVote for 9c64fba479c5e94 [logterm: 1, index: 3] at term 2

  14. Nov 07 09:28:54 node01.k8s.com etcd[1546]: raft.node: 3b8b38de05e2c497 elected leader 9c64fba479c5e94 at term 2

  15. Nov 07 09:28:54 node01.k8s.com etcd[1546]: published {Name:etcd1 ClientURLs:[http://192.168.1.91:2379]} to cluster 19456f0bfd57284e

  16. Nov 07 09:28:54 node01.k8s.com etcd[1546]: ready to serve client requests

  17. Nov 07 09:28:54 node01.k8s.com etcd[1546]: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!

  18. Nov 07 09:28:54 node01.k8s.com systemd[1]: Started Etcd Server.

  19. Nov 07 09:28:54 node01.k8s.com etcd[1546]: set the initial cluster version to 3.3

  20. Nov 07 09:28:54 node01.k8s.com etcd[1546]: enabled capabilities for version 3.3

查看 /var/log/message 日志中,会有如下信息:

 
   
   
 
  1. Nov 7 09:28:53 node02 etcd: added member 9c64fba479c5e94 [http://192.168.1.92:2380] to cluster 19456f0bfd57284e

  2. Nov 7 09:28:53 node02 etcd: added member 3b8b38de05e2c497 [http://192.168.1.91:2380] to cluster 19456f0bfd57284e

  3. Nov 7 09:28:53 node02 etcd: added member 76ea8679db7365b3 [http://192.168.1.93:2380] to cluster 19456f0bfd57284e

查看集群状态:

 
   
   
 
  1. [root@node01 etcd]# ETCDCTL_API=3 etcdctl --endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 endpoint health

  2. http://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.103545ms

  3. http://192.168.1.93:2379 is healthy: successfully committed proposal: took = 2.122478ms

  4. http://192.168.1.91:2379 is healthy: successfully committed proposal: took = 2.690215ms

  5. [root@node01 etcd]# etcdctl --endpoints=http://192.168.1.91:2379,http://192.168.1.92:2379,http://192.168.1.93:2379 cluster-health

  6. member 9c64fba479c5e94 is healthy: got healthy result from http://192.168.1.92:2379

  7. member 3b8b38de05e2c497 is healthy: got healthy result from http://192.168.1.91:2379

  8. member 76ea8679db7365b3 is healthy: got healthy result from http://192.168.1.93:2379

  9. cluster is healthy

生成TLS证书

使用自签证书,CA(Certificate Authority)是自签名的根证书,用来签名后续创建的其他证书。本文使用 CloudFlare 的 PKI 工具 cfssl 创建所有证书。

整个证书的创建过程均在 node01 上操作,首先安装 cfssl 工具集:

 
   
   
 
  1. mkdir -p /opt/k8s/cert && cd /opt/k8s

  2. wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

  3. mv cfssl_linux-amd64 /opt/k8s/bin/cfssl

  4. wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

  5. mv cfssljson_linux-amd64 /opt/k8s/bin/cfssljson

  6. wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

  7. mv cfssl-certinfo_linux-amd64 /opt/k8s/bin/cfssl-certinfo

  8. chmod +x /opt/k8s/bin/*

  9. echo 'export PATH=/opt/k8s/bin:$PATH' >> ~/.bash_profile

  10. source ~/.bash_profile

生成证书

创建根证书 (CA):CA证书是集群所有节点共享的,只需要创建一个CA证书,后续创建的所有证书都是由它签名。

创建配置文件:CA配置文件用于配置根证书的使用场景(profile)和具体参数(usage、过期时间、服务端认证、客户端认证、加密等)

 
   
   
 
  1. cd /opt/k8s/work

  2. cat > ca-config.json <<EOF

  3. {

  4. "signing": {

  5. "default": {

  6. "expiry": "87600h"

  7. },

  8. "profiles": {

  9. "kubernetes": {

  10. "usages": [

  11. "signing",

  12. "key encipherment",

  13. "server auth",

  14. "client auth"

  15. ],

  16. "expiry": "87600h"

  17. }

  18. }

  19. }

  20. }

  21. EOF

signing 表示该证书可用于签名其它证书,生成的 ca.pem 证书找中 CA=TRUE,server auth 表示client可以用该证书对server提供的证书进行验证,client auth 表示server可以用该证书对client提供的证书进行验证

创建证书签名请求文件

 
   
   
 
  1. cd /opt/k8s/work

  2. cat > ca-csr.json <<EOF

  3. {

  4. "CN": "kubernetes",

  5. "key": {

  6. "algo": "rsa",

  7. "size": 2048

  8. },

  9. "names": [

  10. {

  11. "C": "CN",

  12. "ST": "BeiJing",

  13. "L": "BeiJing",

  14. "O": "k8s",

  15. "OU": "4Paradigm"

  16. }

  17. ],

  18. "ca": {

  19. "expiry": "876000h"

  20. }

  21. }

  22. EOF

CN CommonName,kube-apiserver从证书中提取该字段作为请求的用户名(User Name),浏览器使用该字段验证网站是否合法;O Organization,kube-apiserver 从证书中提取该字段作为请求用户和所属组(Group),kube-apiserver将提取的User、Group作为RBAC授权的用户和标识

生成CA证书和私钥

 
   
   
 
  1. cd /opt/k8s/work

  2. cfssl gencert -initca ca-csr.json | cfssljson -bare ca

  3. ls ca*

创建etcd证书和私钥

 
   
   
 
  1. cd /opt/k8s/work

  2. cat > etcd-csr.json <<EOF

  3. {

  4. "CN": "etcd",

  5. "hosts": [

  6. "127.0.0.1",

  7. "192.168.1.91",

  8. "192.168.1.92",

  9. "192.168.1.93",

  10. "k8s.com",

  11. "etcd1.k8s.com",

  12. "etcd2.k8s.com",

  13. "etcd3.k8s.com"

  14. ],

  15. "key": {

  16. "algo": "rsa",

  17. "size": 2048

  18. },

  19. "names": [

  20. {

  21. "C": "CN",

  22. "ST": "BeiJing",

  23. "L": "BeiJing",

  24. "O": "k8s",

  25. "OU": "4Paradigm"

  26. }

  27. ]

  28. }

  29. EOF

host 字段指定授权使用该证书的etcd节点IP或域名列表,需要将etcd集群的3个节点都添加其中 生成证书和私钥

 
   
   
 
  1. cd /opt/k8s/work


  2. cfssl gencert -ca=/opt/k8s/work/ca.pem \

  3. -ca-key=/opt/k8s/work/ca-key.pem \

  4. -config=/opt/k8s/work/ca-config.json \

  5. -profile=kubernetes etcd-csr.json | cfssljson -bare etcd


  6. ls etcd*pem -l

  7. -rw------- 1 root root 1675 Nov 7 09:52 etcd-key.pem

  8. -rw-r--r-- 1 root root 1444 Nov 7 09:52 etcd.pem

etcd 使用的TLS证书创建完成。然后分发证书到各节点上,要做所有节点上创建对应的目录:

 
   
   
 
  1. mkdir /data/k8s/etcd/{data,wal} -p

  2. mkdir -p /etc/kubernetes/cert

  3. chown -R etcd.etcd /data/k8s/etcd

分发证书:

 
   
   
 
  1. cd /opt/k8s/work

  2. scp ca*.pem ca-config.json 192.168.1.91:/etc/kubernetes/cert

  3. scp ca*.pem ca-config.json 192.168.1.92:/etc/kubernetes/cert

  4. scp ca*.pem ca-config.json 192.168.1.93:/etc/kubernetes/cert

  5. scp etcd*pem 192.168.1.91:/etc/etcd/cert/

  6. scp etcd*pem 192.168.1.92:/etc/etcd/cert/

  7. scp etcd*pem 192.168.1.93:/etc/etcd/cert/

在所有节点上执行:

 
   
   
 
  1. chown -R etcd.etcd /etc/etcd/cert

静态TLS集群

node01 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="https://192.168.1.91:2380"

  4. ETCD_LISTEN_CLIENT_URLS="https://192.168.1.91:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd1"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.91:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.91:2379"

  13. ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"


  16. ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

  17. ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  18. ETCD_CLIENT_CERT_AUTH="true"

  19. ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  20. ETCD_AUTO_TLS="true"

  21. ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

  22. ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  23. ETCD_PEER_CLIENT_CERT_AUTH="true"

  24. ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  25. ETCD_PEER_AUTO_TLS="true"

node02 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"

  4. ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd2"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.92:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.92:2379"

  13. ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"


  16. ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

  17. ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  18. ETCD_CLIENT_CERT_AUTH="true"

  19. ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  20. ETCD_AUTO_TLS="true"

  21. ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

  22. ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  23. ETCD_PEER_CLIENT_CERT_AUTH="true"

  24. ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  25. ETCD_PEER_AUTO_TLS="true"

node03 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"

  4. ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd3"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.93:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.93:2379"

  13. ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.91:2380,etcd2=https://192.168.1.92:2380,etcd3=https://192.168.1.93:2380"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"


  16. ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

  17. ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  18. ETCD_CLIENT_CERT_AUTH="true"

  19. ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  20. ETCD_AUTO_TLS="true"

  21. ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

  22. ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  23. ETCD_PEER_CLIENT_CERT_AUTH="true"

  24. ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  25. ETCD_PEER_AUTO_TLS="true"

启动测试:

 
   
   
 
  1. [root@node01 work]# systemctl start etcd

  2. [root@node01 work]# systemctl status etcd

  3. etcd.service - Etcd Server

  4. Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)

  5. Active: active (running) since Thu 2019-11-07 10:15:58 CST; 5s ago

  6. Main PID: 2078 (etcd)

  7. Tasks: 8

  8. Memory: 28.9M

  9. CGroup: /system.slice/etcd.service

  10. └─2078 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=https://192.168.1.91:2379


  11. Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe [term: 1] received a MsgVote message with higher term from af05139f75a68867 [term: 2]

  12. Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe became follower at term 2

  13. Nov 07 10:15:58 node01.k8s.com etcd[2078]: 2a40d8ba966d12fe [logterm: 1, index: 3, vote: 0] cast MsgVote for af05139f75a68867 [logterm: 1, index: 3] at term 2

  14. Nov 07 10:15:58 node01.k8s.com etcd[2078]: raft.node: 2a40d8ba966d12fe elected leader af05139f75a68867 at term 2

  15. Nov 07 10:15:58 node01.k8s.com etcd[2078]: published {Name:etcd1 ClientURLs:[https://192.168.1.91:2379]} to cluster f3e9c54e1aafb3c1

  16. Nov 07 10:15:58 node01.k8s.com etcd[2078]: ready to serve client requests

  17. Nov 07 10:15:58 node01.k8s.com etcd[2078]: serving client requests on 192.168.1.91:2379

  18. Nov 07 10:15:58 node01.k8s.com systemd[1]: Started Etcd Server.

  19. Nov 07 10:15:58 node01.k8s.com etcd[2078]: set the initial cluster version to 3.3

  20. Nov 07 10:15:58 node01.k8s.com etcd[2078]: enabled capabilities for version 3.3

查看 /var/log/message 日志中,会有日下体现:

 
   
   
 
  1. Nov 7 10:15:57 node01 etcd: added member 2a40d8ba966d12fe [https://192.168.1.91:2380] to cluster f3e9c54e1aafb3c1

  2. Nov 7 10:15:57 node01 etcd: added member af05139f75a68867 [https://192.168.1.92:2380] to cluster f3e9c54e1aafb3c1

  3. Nov 7 10:15:57 node01 etcd: added member c3bab7c20fba3f60 [https://192.168.1.93:2380] to cluster f3e9c54e1aafb3c1

检查TLS集群状态:

 
   
   
 
  1. ETCDCTL_API=3 etcdctl \

  2. --endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \

  3. --cacert=/etc/kubernetes/cert/ca.pem \

  4. --cert=/etc/etcd/cert/etcd.pem \

  5. --key=/etc/etcd/cert/etcd-key.pem endpoint health


  6. # 输出

  7. https://192.168.1.92:2379 is healthy: successfully committed proposal: took = 1.317022ms

  8. https://192.168.1.91:2379 is healthy: successfully committed proposal: took = 1.59958ms

  9. https://192.168.1.93:2379 is healthy: successfully committed proposal: took = 1.453049ms

ETCD 动态集群基于 DNS 的 SRV 解析自动发现

需要局域网内部有DNS服务器

添加SRV解析

目前常用的内部DNS服务有两种,bind、dnsmasq。

在下面都会列出具体的配置,但只需要配置其中之一即可;

方法一: 使用 bind 配置SRV解析 如果内部没有 bind 服务,可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11806962.html

使用域名为: k8s.com,在 bind 的 zone 文件中添加如下解析:

 
   
   
 
  1. etcd1 IN A 192.168.1.91

  2. etcd2 IN A 192.168.1.92

  3. etcd3 IN A 192.168.1.93

  4. _etcd-server._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd1

  5. _etcd-server._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd2

  6. _etcd-server._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd3

  7. _etcd-client._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd1

  8. _etcd-client._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd2

  9. _etcd-client._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd3

修改之后重新加载配置文件:

 
   
   
 
  1. [root@jenkins named]# named-checkzone k8s.com k8s.com.zone

  2. zone k8s.com/IN: loaded serial 0

  3. OK

  4. [root@jenkins named]# rndc reload

  5. server reload successful

方法二: 使用 dnsmasq配置SRV解析如果内部没有dnsmasq 服务,可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11809066.html

使用域名为: k8s.com,具体修改如下,首先在 /etc/dnsmasq_hosts新增下面内容:

 
   
   
 
  1. 192.168.1.91 etcd1 etcd1.k8s.com

  2. 192.168.1.92 etcd2 etcd2.k8s.com

  3. 192.168.1.93 etcd3 etcd3.k8s.com

/etc/dnsmasq.conf 文件中增加下面 SRV 解析内容:

 
   
   
 
  1. srv-host=_etcd-server._tcp.k8s.com,etcd1.k8s.com,2380,0,100

  2. srv-host=_etcd-server._tcp.k8s.com,etcd2.k8s.com,2380,0,100

  3. srv-host=_etcd-server._tcp.k8s.com,etcd3.k8s.com,2380,0,100

  4. srv-host=_etcd-client._tcp.k8s.com,etcd1.k8s.com,2380,0,100

  5. srv-host=_etcd-client._tcp.k8s.com,etcd2.k8s.com,2380,0,100

  6. srv-host=_etcd-client._tcp.k8s.com,etcd3.k8s.com,2380,0,100

修改之后重启服务:

 
   
   
 
  1. systemctl restart dnsmasq

验证 SRV 解析是否正常,查询 SRV 记录:

 
   
   
 
  1. [root@node01 ~]# dig @192.168.1.122 +noall +answer SRV _etcd-server._tcp.k8s.com

  2. _etcd-server._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd2.k8s.com.

  3. _etcd-server._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd1.k8s.com.

  4. _etcd-server._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd3.k8s.com.

查询域名解析结果:

 
   
   
 
  1. [root@node01 ~]# dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com

  2. etcd1.k8s.com. 86400 IN A 192.168.1.91

  3. etcd2.k8s.com. 86400 IN A 192.168.1.92

  4. etcd3.k8s.com. 86400 IN A 192.168.1.93

如上述显示,则表示 SRV 解析正常。

配置ETCD

node01 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="http://192.168.1.91:2380"

  4. ETCD_LISTEN_CLIENT_URLS="http://192.168.1.91:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd1"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd1.k8s.com:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="http://etcd1.k8s.com:2379"

  13. ETCD_DISCOVERY_SRV="k8s.com"


  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"

node02 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="http://192.168.1.92:2380"

  4. ETCD_LISTEN_CLIENT_URLS="http://192.168.1.92:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd2"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd2.k8s.com:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="http://etcd2.k8s.com:2379"

  13. ETCD_DISCOVERY_SRV="k8s.com"


  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"

node03 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="http://192.168.1.93:2380"

  4. ETCD_LISTEN_CLIENT_URLS="http://192.168.1.93:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd3"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="http://etcd3.k8s.com:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="http://etcd3.k8s.com:2379"

  13. ETCD_DISCOVERY_SRV="k8s.com"


  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"

启动并测试:

 
   
   
 
  1. [root@node01 etcd]# systemctl start etcd

  2. [root@node01 etcd]# systemctl status etcd

  3. etcd.service - Etcd Server

  4. Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)

  5. Active: active (running) since Thu 2019-11-07 11:25:29 CST; 4s ago

  6. Main PID: 14203 (etcd)

  7. Tasks: 8

  8. Memory: 16.9M

  9. CGroup: /system.slice/etcd.service

  10. └─14203 /usr/bin/etcd --name=etcd1 --data-dir=/data/k8s/etcd/data --listen-client-urls=http://192.168.1.91:2379


  11. Nov 07 11:25:29 node01.k8s.com etcd[14203]: d79e9ae86b2a1de1 [quorum:2] has received 2 MsgVoteResp votes and 0 vote rejections

  12. Nov 07 11:25:29 node01.k8s.com etcd[14203]: d79e9ae86b2a1de1 became leader at term 2

  13. Nov 07 11:25:29 node01.k8s.com etcd[14203]: raft.node: d79e9ae86b2a1de1 elected leader d79e9ae86b2a1de1 at term 2

  14. Nov 07 11:25:29 node01.k8s.com etcd[14203]: published {Name:etcd1 ClientURLs:[http://etcd1.k8s.com:2379 http://etcd1.k8s.com:4001]} to cluster 42cecf80e3791d6c

  15. Nov 07 11:25:29 node01.k8s.com etcd[14203]: ready to serve client requests

  16. Nov 07 11:25:29 node01.k8s.com etcd[14203]: serving insecure client requests on 192.168.1.91:2379, this is strongly discouraged!

  17. Nov 07 11:25:29 node01.k8s.com systemd[1]: Started Etcd Server.

  18. Nov 07 11:25:29 node01.k8s.com etcd[14203]: setting up the initial cluster version to 3.3

  19. Nov 07 11:25:29 node01.k8s.com etcd[14203]: set the initial cluster version to 3.3

  20. Nov 07 11:25:29 node01.k8s.com etcd[14203]: enabled capabilities for version 3.3

日志 vim /var/log/messages 表现如下:

 
   
   
 
  1. Nov 7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at 0=http://etcd3.k8s.com:2380

  2. Nov 7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at 1=http://etcd2.k8s.com:2380

  3. Nov 7 11:25:27 node01 etcd: got bootstrap from DNS for etcd-server at etcd1=http://etcd1.k8s.com:2380

  4. Nov 7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380

  5. Nov 7 11:25:27 node01 etcd: resolving etcd1.k8s.com:2380 to 192.168.1.91:2380

  6. Nov 7 11:25:28 node01 etcd: name = etcd1

  7. Nov 7 11:25:28 node01 etcd: data dir = /data/k8s/etcd/data

  8. Nov 7 11:25:28 node01 etcd: member dir = /data/k8s/etcd/data/member

  9. Nov 7 11:25:28 node01 etcd: dedicated WAL dir = /data/k8s/etcd/wal

  10. Nov 7 11:25:28 node01 etcd: heartbeat = 100ms

  11. Nov 7 11:25:28 node01 etcd: election = 1000ms

  12. Nov 7 11:25:28 node01 etcd: snapshot count = 100000

  13. Nov 7 11:25:28 node01 etcd: advertise client URLs = http://etcd1.k8s.com:2379,http://etcd1.k8s.com:4001

  14. Nov 7 11:25:28 node01 etcd: initial advertise peer URLs = http://etcd1.k8s.com:2380

  15. Nov 7 11:25:28 node01 etcd: initial cluster = 0=http://etcd3.k8s.com:2380,1=http://etcd2.k8s.com:2380,etcd1=http://etcd1.k8s.com:2380

测试:

 
   
   
 
  1. [root@node01 etcd]# etcdctl --endpoints=http://192.168.1.91:2379 cluster-health

  2. member 184beca37ca32d75 is healthy: got healthy result from http://etcd2.k8s.com:2379

  3. member d79e9ae86b2a1de1 is healthy: got healthy result from http://etcd1.k8s.com:2379

  4. member f7662e609b7e4013 is healthy: got healthy result from http://etcd3.k8s.com:2379

  5. cluster is healthy

到这里证明已经成功了。

ETCD TLS 动态集群基于 DNS 的 SRV 解析自动发现

同样需要局域网内部有 DNS 服务器

添加SRV解析

目前常用的内部DNS服务有两种,bind、dnsmasq

在下面都会列出具体的配置,但只需要配置其中之一即可;

方法一: 使用bind配置SRV解析 如果内部没有bind服务,可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11806962.html

使用域名为: k8s.com,在 bind 的 zone 文件中添加如下解析:

 
   
   
 
  1. etcd1 IN A 192.168.1.91

  2. etcd2 IN A 192.168.1.92

  3. etcd3 IN A 192.168.1.93

  4. _etcd-server-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd1

  5. _etcd-server-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd2

  6. _etcd-server-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd3

  7. _etcd-client-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd1

  8. _etcd-client-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd2

  9. _etcd-client-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd3

修改之后重新加载配置文件:

 
   
   
 
  1. [root@jenkins named]# named-checkzone k8s.com k8s.com.zone

  2. zone k8s.com/IN: loaded serial 0

  3. OK

  4. [root@jenkins named]# rndc reload

  5. server reload successful

方法二: 使用 dnsmasq 配置 SRV 解析 如果内部没有 dnsmasq 服务,可以参考部署文档文章: https://www.cnblogs.com/winstom/p/11809066.html

使用域名为: k8s.com,具体修改如下,在 /etc/dnsmasq_hosts新增下面内容:

 
   
   
 
  1. 192.168.1.91 etcd1 etcd1.k8s.com

  2. 192.168.1.92 etcd2 etcd2.k8s.com

  3. 192.168.1.93 etcd3 etcd3.k8s.com

/etc/dnsmasq.conf 文件中增加下面 SRV 解析内容:

 
   
   
 
  1. srv-host=_etcd-server-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100

  2. srv-host=_etcd-server-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100

  3. srv-host=_etcd-server-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100

  4. srv-host=_etcd-client-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100

  5. srv-host=_etcd-client-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100

  6. srv-host=_etcd-client-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100

修改之后重启服务:

 
   
   
 
  1. systemctl restart dnsmasq

验证 SRV 解析是否正常,查询 SRV记录:

 
   
   
 
  1. [root@node01 etcd]# dig @192.168.1.122 +noall +answer SRV _etcd-server-ssl._tcp.k8s.com

  2. _etcd-server-ssl._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd3.k8s.com.

  3. _etcd-server-ssl._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd2.k8s.com.

  4. _etcd-server-ssl._tcp.k8s.com. 3600 IN SRV 2380 0 100 etcd1.k8s.com.

查询域名解析结果:

 
   
   
 
  1. [root@node01 ~]# dig @192.168.1.122 +noall +answer etcd1.k8s.com etcd2.k8s.com etcd3.k8s.com

  2. etcd1.k8s.com. 86400 IN A 192.168.1.91

  3. etcd2.k8s.com. 86400 IN A 192.168.1.92

  4. etcd3.k8s.com. 86400 IN A 192.168.1.93

ETCD 配置

node01 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="https://192.168.1.91:2380"

  4. ETCD_LISTEN_CLIENT_URLS="https://192.168.1.91:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd1"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd1.k8s.com:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="https://etcd1.k8s.com:2379,https://etcd1.k8s.com:4001"

  13. ETCD_DISCOVERY_SRV="k8s.com"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"


  16. ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

  17. ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  18. ETCD_CLIENT_CERT_AUTH="true"

  19. ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  20. ETCD_AUTO_TLS="true"

  21. ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

  22. ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  23. ETCD_PEER_CLIENT_CERT_AUTH="true"

  24. ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  25. ETCD_PEER_AUTO_TLS="true"

node02 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="https://192.168.1.92:2380"

  4. ETCD_LISTEN_CLIENT_URLS="https://192.168.1.92:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd2"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd2.k8s.com:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="https://etcd2.k8s.com:2379"

  13. ETCD_DISCOVERY_SRV="k8s.com"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"


  16. ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

  17. ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  18. ETCD_CLIENT_CERT_AUTH="true"

  19. ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  20. ETCD_AUTO_TLS="true"

  21. ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

  22. ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  23. ETCD_PEER_CLIENT_CERT_AUTH="true"

  24. ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  25. ETCD_PEER_AUTO_TLS="true"

node03 配置文件:

 
   
   
 
  1. ETCD_DATA_DIR="/data/k8s/etcd/data"

  2. ETCD_WAL_DIR="/data/k8s/etcd/wal"

  3. ETCD_LISTEN_PEER_URLS="https://192.168.1.93:2380"

  4. ETCD_LISTEN_CLIENT_URLS="https://192.168.1.93:2379"

  5. ETCD_MAX_SNAPSHOTS="5"

  6. ETCD_MAX_WALS="5"

  7. ETCD_NAME="etcd3"

  8. ETCD_SNAPSHOT_COUNT="100000"

  9. ETCD_HEARTBEAT_INTERVAL="100"

  10. ETCD_ELECTION_TIMEOUT="1000"


  11. ETCD_INITIAL_ADVERTISE_PEER_URLS="https://etcd3.k8s.com:2380"

  12. ETCD_ADVERTISE_CLIENT_URLS="https://etcd3.k8s.com:2379"

  13. ETCD_DISCOVERY_SRV="k8s.com"

  14. ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"

  15. ETCD_INITIAL_CLUSTER_STATE="new"


  16. ETCD_CERT_FILE="/etc/etcd/cert/etcd.pem"

  17. ETCD_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  18. ETCD_CLIENT_CERT_AUTH="true"

  19. ETCD_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  20. ETCD_AUTO_TLS="true"

  21. ETCD_PEER_CERT_FILE="/etc/etcd/cert/etcd.pem"

  22. ETCD_PEER_KEY_FILE="/etc/etcd/cert/etcd-key.pem"

  23. ETCD_PEER_CLIENT_CERT_AUTH="true"

  24. ETCD_PEER_TRUSTED_CA_FILE="/etc/kubernetes/cert/ca.pem"

  25. ETCD_PEER_AUTO_TLS="true"

启动测试:

 
   
   
 
  1. [root@node03 etcd]# systemctl restart etcd

  2. [root@node03 etcd]# systemctl status etcd

  3. etcd.service - Etcd Server

  4. Loaded: loaded (/usr/lib/systemd/system/etcd.service; disabled; vendor preset: disabled)

  5. Active: active (running) since Thu 2019-11-07 12:38:37 CST; 4s ago

  6. Main PID: 13460 (etcd)

  7. Tasks: 8

  8. Memory: 16.6M

  9. CGroup: /system.slice/etcd.service

  10. └─13460 /usr/bin/etcd --name=etcd3 --data-dir=/data/k8s/etcd/data --listen-client-urls=https://192.168.1.93:2379


  11. Nov 07 12:38:36 node03.k8s.com etcd[13460]: established a TCP streaming connection with peer 40a8f19a5db99534 (stream Message writer)

  12. Nov 07 12:38:36 node03.k8s.com etcd[13460]: established a TCP streaming connection with peer 40a8f19a5db99534 (stream MsgApp v2 writer)

  13. Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 [term: 92] received a MsgVote message with higher term from a0d541999e9eb3b3 [term: 98]

  14. Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 became follower at term 98

  15. Nov 07 12:38:37 node03.k8s.com etcd[13460]: 9888555207dbf0e0 [logterm: 92, index: 9, vote: 0] cast MsgVote for a0d541999e9eb3b3 [logterm: 92, index: 9] at term 98

  16. Nov 07 12:38:37 node03.k8s.com etcd[13460]: raft.node: 9888555207dbf0e0 elected leader a0d541999e9eb3b3 at term 98

  17. Nov 07 12:38:37 node03.k8s.com etcd[13460]: published {Name:etcd3 ClientURLs:[https://etcd3.k8s.com:2379]} to cluster f445a02ce3dc6a02

  18. Nov 07 12:38:37 node03.k8s.com etcd[13460]: ready to serve client requests

  19. Nov 07 12:38:37 node03.k8s.com etcd[13460]: serving client requests on 192.168.1.93:2379

  20. Nov 07 12:38:37 node03.k8s.com systemd[1]: Started Etcd Server.

查看 message 日志:

 
   
   
 
  1. Nov 7 12:38:36 node01 etcd: added member 40a8f19a5db99534 [https://etcd2.k8s.com:2380] to cluster f445a02ce3dc6a02

  2. Nov 7 12:38:36 node01 etcd: starting peer 40a8f19a5db99534...

  3. Nov 7 12:38:36 node01 etcd: started HTTP pipelining with peer 40a8f19a5db99534

  4. Nov 7 12:38:36 node01 etcd: started streaming with peer 40a8f19a5db99534 (writer)

  5. Nov 7 12:38:36 node01 etcd: started peer 40a8f19a5db99534

  6. Nov 7 12:38:36 node01 etcd: added peer 40a8f19a5db99534

  7. Nov 7 12:38:36 node01 etcd: added member 9888555207dbf0e0 [https://etcd3.k8s.com:2380] to cluster f445a02ce3dc6a02

  8. Nov 7 12:38:36 node01 etcd: starting peer 9888555207dbf0e0...

  9. Nov 7 12:38:36 node01 etcd: started HTTP pipelining with peer 9888555207dbf0e0

  10. Nov 7 12:38:36 node01 etcd: started peer 9888555207dbf0e0

  11. Nov 7 12:38:36 node01 etcd: added peer 9888555207dbf0e0

  12. Nov 7 12:38:36 node01 etcd: added member a0d541999e9eb3b3 [https://etcd1.k8s.com:2380] to cluster f445a02ce3dc6a02

测试集群状态:

 
   
   
 
  1. ETCDCTL_API=3 etcdctl --endpoints=https://etcd1.k8s.com:2379,https://etcd2.k8s.com:2379,https://etcd3.k8s.com:2379 \

  2. --cacert=/etc/kubernetes/cert/ca.pem \

  3. --cert=/etc/etcd/cert/etcd.pem \

  4. --key=/etc/etcd/cert/etcd-key.pem endpoint health


  5. # 输出

  6. https://etcd1.k8s.com:2379 is healthy: successfully committed proposal: took = 4.269468ms

  7. https://etcd3.k8s.com:2379 is healthy: successfully committed proposal: took = 1.58797ms

  8. https://etcd2.k8s.com:2379 is healthy: successfully committed proposal: took = 1.622151ms

报错解决

  1. 证书报错 bad certificate,日志中报错:

 
   
   
 
  1. Nov 7 12:37:03 node01 etcd: rejected connection from "192.168.1.93:46294" (error "remote error: tls: bad certificate", ServerName "k8s.com")

报错的意思是在生成 ETCD 的 TLS 证书的时候,没有把对应的域名加进去,在创建 ETCD 的 TLS 证书请求的文件中加入对应的域名:

 
   
   
 
  1. cd /opt/k8s/work

  2. cat > etcd-csr.json <<EOF

  3. {

  4. "CN": "etcd",

  5. "hosts": [

  6. "127.0.0.1",

  7. "192.168.1.91",

  8. "192.168.1.92",

  9. "192.168.1.93",

  10. "k8s.com", # 这里的域名查看是否正确

  11. "etcd1.k8s.com",

  12. "etcd2.k8s.com",

  13. "etcd3.k8s.com"

  14. ],

  15. "key": {

  16. "algo": "rsa",

  17. "size": 2048

  18. },

  19. "names": [

  20. {

  21. "C": "CN",

  22. "ST": "BeiJing",

  23. "L": "BeiJing",

  24. "O": "k8s",

  25. "OU": "4Paradigm"

  26. }

  27. ]

  28. }

  29. EOF

  1. DNS 的 SRV 解析报错:

 
   
   
 
  1. etcd: error setting up initial cluster: cannot find local etcd member "etcd1" in SRV records

这里是说 DNS 在配置 SRV 解析的时候报错,请仔细查看解析配置,SRV 解析分为两种,一种是 http 不带证书的解析,一种是 https 带证书的解析,是有区别的,如果配置错误就会包上述错误

http不带证书解析如下:

bind 的解析,编辑 /var/named/k8s.com.zone 文件

 
   
   
 
  1. etcd1 IN A 192.168.1.91

  2. etcd2 IN A 192.168.1.92

  3. etcd3 IN A 192.168.1.93

  4. _etcd-server._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd1

  5. _etcd-server._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd2

  6. _etcd-server._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd3

  7. _etcd-client._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd1

  8. _etcd-client._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd2

  9. _etcd-client._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd3

dnsmasq 的解析,在/etc/dnsmasq_hosts新增下面内容:

 
   
   
 
  1. 192.168.1.91 etcd1 etcd1.k8s.com

  2. 192.168.1.92 etcd2 etcd2.k8s.com

  3. 192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面SRV解析内容

 
   
   
 
  1. srv-host=_etcd-server._tcp.k8s.com,etcd1.k8s.com,2380,0,100

  2. srv-host=_etcd-server._tcp.k8s.com,etcd2.k8s.com,2380,0,100

  3. srv-host=_etcd-server._tcp.k8s.com,etcd3.k8s.com,2380,0,100

  4. srv-host=_etcd-client._tcp.k8s.com,etcd1.k8s.com,2380,0,100

  5. srv-host=_etcd-client._tcp.k8s.com,etcd2.k8s.com,2380,0,100

  6. srv-host=_etcd-client._tcp.k8s.com,etcd3.k8s.com,2380,0,100

https带证书解析如下:

bind 的解析,编辑 /var/named/k8s.com.zone 文件:

 
   
   
 
  1. etcd1 IN A 192.168.1.91

  2. etcd2 IN A 192.168.1.92

  3. etcd3 IN A 192.168.1.93

  4. _etcd-server-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd1

  5. _etcd-server-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd2

  6. _etcd-server-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd3

  7. _etcd-client-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd1

  8. _etcd-client-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd2

  9. _etcd-client-ssl._tcp.k8s.com. 1H IN SRV 2380 0 100 etcd3

dnsmasq 的解析,在/etc/dnsmasq_hosts新增下面内容:

 
   
   
 
  1. 192.168.1.91 etcd1 etcd1.k8s.com

  2. 192.168.1.92 etcd2 etcd2.k8s.com

  3. 192.168.1.93 etcd3 etcd3.k8s.com

在 /etc/dnsmasq.conf 文件中增加下面 SRV 解析内容:

 
   
   
 
  1. srv-host=_etcd-server-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100

  2. srv-host=_etcd-server-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100

  3. srv-host=_etcd-server-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100

  4. srv-host=_etcd-client-ssl._tcp.k8s.com,etcd1.k8s.com,2380,0,100

  5. srv-host=_etcd-client-ssl._tcp.k8s.com,etcd2.k8s.com,2380,0,100

  6. srv-host=_etcd-client-ssl._tcp.k8s.com,etcd3.k8s.com,2380,0,100

原文链接:https://www.cnblogs.com/winstom/p/11811373.html

作者介绍:技术男一枚,喜欢做技术分享,把学习的过程,以及遇到问题的解决过程都愿意分享给大家,博客中如有不足,请留言或者联系博主,感谢。邮箱:sijiayong000@163.com QQ:601566386





K8S进阶课程推荐

以上是关于CentOS 7 ETCD 集群配置大全的主要内容,如果未能解决你的问题,请参考以下文章

Centos7下Etcd集群搭建

centos7部署etcd集群

Etcd集群安装配置

django + etcd + confd配置管理平台

Centos 7.5私有域名服务器部署(coredns+etcd)

centos 7 etcd服务启动出现see systemcl status etcd.service