部署Flannel，实现跨主机Docker容器通信

Posted 2020-10-15

flannel(flannel is a virtual network that attaches IP addresses to containers)

两个主机下的Docker容器之间是不互通的，通过部署flannel，对docker容器的ip进行规划，就能实现跨主机容器之间的通信。
官方文档：https://coreos.com/flannel/docs/latest/flannel-config.html
以下介绍Flannel的部署
机器配置如下

hostname	ip	系统版本
etcd1	172.17.0.2	CentOS Linux release 7.4.1708 (Core)
etcd2	172.17.0.3	CentOS Linux release 7.4.1708 (Core)

一、搭建etcd集群

参见之前的文章：http://www.cnblogs.com/cs-zh/p/7878019.html

二、添加网络配置到etcd

etcdctl --endpoint="172.17.0.2:2379" set /coreos.com/network/config ‘{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}‘

这里配置flannel网络ip段为172.17.0.0
flannel默认加载etcd配置前缀为“/coreos.com/network”，可修改为其他的key

三、下载二进制包

wget https://github.com/coreos/flannel/releases/download/v0.9.1/flannel-v0.9.1-linux-amd64.tar.gz
tar -xf flannel-v0.9.1-linux-amd64.tar.gz

四、启动flannel

nohup ./flanneld >> flannel.log 2>&1 &

注意：1.如果“第二步”中修改etcdctl set的key值，这里需要加参数-etcd-prefix=".."
2.如果etcd不在部署flannel的节点，需要加参数-etcd-endpoints
具体./flanneld -h看参数说明
运行成功后，会生成文件/run/flannel/subnet.env，之后需要重启dockerd，要带上这几个信息

FLANNEL_NETWORK=172.17.0.0/16
FLANNEL_SUBNET=172.17.80.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

五、重启dockerd

1.修改docker.service

vim /usr/lib/systemd/system/docker.service

增加EnvironmentFile，并在ExecStart后面加上"$DOCKER_OPTS"

EnvironmentFile=/etc/default/docker
ExecStart=/usr/bin/dockerd $DOCKER_OPTS

2.重新加载配置

systemctl daemon-reload

3.重启dockerd

systemctl restart docker

4.确认参数正确执行

ps -ef|grep docker

查看进行信息，--bip参数已经被正确带上

root      4713     1  0 07:48 ?        00:00:00 /usr/bin/dockerd --bip=172.17.80.1/24 --ip-masq=true --mtu=1450
root      4721  4713  0 07:48 ?        00:00:00 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc

六、确认配置是否正确

1.查看网卡信息

ifconfig

此时多出个“flannel.1”，并且docker0网卡在该子网下

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.80.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 02:42:2d:68:34:d8  txqueuelen 0  (Ethernet)
        RX packets 568  bytes 45301 (44.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 565  bytes 66158 (64.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.17.0.2  netmask 255.255.0.0  broadcast 0.0.0.0
        ether 02:42:ac:11:00:02  txqueuelen 0  (Ethernet)
        RX packets 983380  bytes 612479051 (584.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 964071  bytes 89535273 (85.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.17.80.0  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 2e:3c:6e:38:36:ce  txqueuelen 0  (Ethernet)
        RX packets 19  bytes 1464 (1.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 21  bytes 1593 (1.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

2.查看路由配置

route -n

确认有flannel.1的路由表

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.17.0.1      0.0.0.0         UG    0      0        0 eth0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth0
172.17.80.0     0.0.0.0         255.255.255.0   U     0      0        0 docker0
172.17.87.0     172.17.87.0     255.255.255.0   UG    0      0        0 flannel.1

假如这里没有flannel.1的路由表，则跨主机容器之间还是ping不同，因为数据包没有被正确的转发。看看是不是flannel的版本过低，这个问题在0.8.0出现，换成最新0.9.1就没问题。

七.在其他节点继续按照“三”到“六”的步骤做一遍

八.验证是否能ping通

1.在节点1运行

docker run -it --rm=true busybox sh

进入容器后执行

ifconfig

此容器的ip是172.17.80.2

eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:50:02  
          inet addr:172.17.80.2  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

2.同样在节点2启动容器并查看ip：“172.17.87.2”
在此容器上执行

ping 172.17.80.2

PING 172.17.80.2 (172.17.80.2): 56 data bytes
64 bytes from 172.17.80.2: seq=0 ttl=62 time=0.415 ms
64 bytes from 172.17.80.2: seq=1 ttl=62 time=0.214 ms