跨主机容器之间通信实现方式：etcd+flanned

Posted 2020-10-02

一、环境说明

node-1: 10.0.0.168
node-2: 10.0.0.169
node-3: 10.0.0.170

二、初始化环境

1、hostnamectl --static set-hostname hostname

2、/etc/hosts

10.0.0.168 node-1
10.0.0.169 node-2
10.0.0.170 node-3

三、安装etcd集群

1、在每台机器上

yum install -y etcd

2、配置 /etc/etcd/etcd.conf

(a)、node1

ETCD_NAME=etcd1
ETCD_DATA_DIR="/var/lib/etcd/etcd1.etcd"
ETCD_LISTEN_PEER_URLS="http://10.0.0.168:2380"
ETCD_LISTEN_CLIENT_URLS="http://10.0.0.168:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.0.168:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://10.0.0.168:2380,etcd2=http://10.0.0.169:2380,etcd3=http://10.0.0.170:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.168:2379"

(2)、node2

ETCD_NAME=etcd2
ETCD_DATA_DIR="/var/lib/etcd/etcd2.etcd"
ETCD_LISTEN_PEER_URLS="http://10.0.0.169:2380"
ETCD_LISTEN_CLIENT_URLS="http://10.0.0.169:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.0.169:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://10.0.0.168:2380,etcd2=http://10.0.0.169:2380,etcd3=http://10.0.0.170:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.169:2379"

(3)、node3

ETCD_NAME=etcd3
ETCD_DATA_DIR="/var/lib/etcd/etcd3.etcd"
ETCD_LISTEN_PEER_URLS="http://10.0.0.170:2380"
ETCD_LISTEN_CLIENT_URLS="http://10.0.0.170:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://10.0.0.170:2380"
ETCD_INITIAL_CLUSTER="etcd1=http://10.0.0.168:2380,etcd2=http://10.0.0.169:2380,etcd3=http://10.0.0.170:2380"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="k8s-etcd-cluster"
ETCD_ADVERTISE_CLIENT_URLS="http://10.0.0.170:2379"

3、修改 etcd 启动文件 /usr/lib/systemd/system/etcd.service

sed -i ‘s/\\\"${ETCD_LISTEN_CLIENT_URLS}\\\"/\\\"${ETCD_LISTEN_CLIENT_URLS}\\\" --listen-client-urls=\\\"${ETCD_LISTEN_CLIENT_URLS}\\\" --advertise-client-urls=\\\"${ETCD_ADVERTISE_CLIENT_URLS}\\\" --initial-cluster-token=\\\"${ETCD_INITIAL_CLUSTER_TOKEN}\\\" --initial-cluster=\\\"${ETCD_INITIAL_CLUSTER}\\\" --initial-cluster-state=\\\"${ETCD_INITIAL_CLUSTER_STATE}\\\"/g‘ /usr/lib/systemd/system/etcd.service

4、分别启动 所有节点的 etcd 服务

systemctl enable etcd
systemctl start etcd
systemctl status etcd

5、查看etcd集群状态，以及集群成员

etcdctl cluster-health
etcdctl member list

四、flanneld网络

1、安装flannel

yum install -y flannel

2、清除网络中遗留的docker网络（如 docker0 flannel0等）

ip link delete docker0

3、设置flannel所用到的网络

etcdctl --endpoint http://10.0.0.168:2379 set /flannel/network/config ‘{"Network":"172.16.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan","VNI":1}}‘
etcdctl --endpoint http://10.0.0.169:2379 set /flannel/network/config ‘{"Network":"172.16.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan","VNI":1}}‘
etcdctl --endpoint http://10.0.0.170:2379 set /flannel/network/config ‘{"Network":"172.16.0.0/16","SubnetLen":24,"Backend":{"Type":"vxlan","VNI":1}}‘

4、修改flannel配置文件(/etc/sysconfig/flanneld)

FLANNEL_ETCD="http://10.0.0.168:2379,http://10.0.0.169:2379,http://10.0.0.170:2379"      # 修改为 集群地址
FLANNEL_ETCD_PREFIX="/flannel/network"                                                   # 修改为 上面导入配置中的  /flannel/network
FLANNEL_OPTIONS="--iface=eth0"  

5、启动flannel

systemctl enable flanneld
systemctl start flanneld
systemctl status flanneld

五、修改docker配置(/usr/lib/systemd/system/docker.service)

1、增加如下配置

ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS

2、重新读取配置，并启动docker

systemctl daemon-reload
systemctl start docker

3、查看网络接口

ifconfig

如果看到 docker0 与 flannel1 已经在我们设置的IP段内了，表示已经成功

4、验证

可以在各个服务器上新建docker容器，然后通过(docker exec test1 ping 172.16.5.*)来验证