OVS+Docker网络打通

Posted 2020-07-26

 

1. 机器列表

主机名 IP docker0 IP docker容器IP
node101 192.168.80.101 10.1.1.1/24 10.1.1.2/24
node102 192.168.80.102 20.1.1.1/24 20.1.1.2/24

 

2. 网络示意图

3. rpm制作
mkdir -p ~/rpmbuild/SOURCES
cp openvswitch-2.5.0.tar.gz ~/rpmbuild/SOURCES

使用源码包中的SPEC文件制作rpm安装包
tar -xf openvswitch-2.5.0.tar.gz
rpmbuild -bb openvswitch-2.5.0/rhel/openvswitch.spec

4. node101
==================
4.1 安装制作的rpm包

yum localinstall ~/rpmbuild/RPMS/x86_64/openvswitch-2.5.0-1.x86_64.rpm

 

4.2 启动openswitch
/etc/init.d/openvswitch start
/etc/init.d/openvswitch status

检查日志输出
tail -100 /var/log/messages

4.3 ovs配置

创建网桥br0
ovs-vsctl add-br br0

把网络设备gre1添加到网桥br0
ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre option:remote_ip=192.168.80.102

添加br0到本地docker0，使得容器流量通过OVS流经tunnel
brctl addif docker0 br0

修改网络设备状态为up
ip link set dev br0 up
ip link set docker0 up

查看网桥和ovs接口
brctl show
ovs-vsctl list-br
ovs-vsctl list-ifaces br0
ovs-vsctl list-ports br0

4.4 防火墙放行icmp
iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited
iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited

4.5 添加到node102主机的docker路由
route add -net 20.1.1.0/24 gw 192.168.80.102

4.6 启动容器
docker run -it mysql bash

查看该容器ip地址
ip addr

 

5. node102
==================

5.1 安装制作的rpm包
yum localinstall ~/rpmbuild/RPMS/x86_64/openvswitch-2.5.0-1.x86_64.rpm

5.2 启动openswitch
/etc/init.d/openvswitch start
/etc/init.d/openvswitch status

检查日志输出
tail -100 /var/log/messages

5.3 OVS配置

创建网桥br0
ovs-vsctl add-br br0

把网络设备gre1添加到网桥br0
ovs-vsctl add-port br0 gre1 -- set interface gre1 type=gre option:remote_ip=192.168.80.101

添加br0到本地docker0，使得容器流量通过OVS流经tunnel
brctl addif docker0 br0

修改网络设备状态为up
ip link set dev br0 up
ip link set docker0 up

查看网桥和ovs接口
brctl show
ovs-vsctl list-br
ovs-vsctl list-ifaces br0
ovs-vsctl list-ports br0

5.4 防火墙放行icmp
iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited
iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited

5.5 添加到node102主机的docker路由
route add -net 10.1.1.0/24 gw 192.168.80.101

5.6 启动容器
docker run -it mysql bash

查看该容器ip地址
ip addr

 

6. 抓包分析
在node101中的docker实例中pingnode102的docker实例IP，抓包分析OVS数据流向

node101
==================
[veth2a3e623] 04:03:59.861136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.1.1.2 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[veth2a3e623] 04:03:59.861986 IP (tos 0x0, ttl 62, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 10.1.1.2: ICMP echo reply, id 24, seq 0, length 64

[docker0] 04:03:59.861136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.1.1.2 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[docker0] 04:03:59.861979 IP (tos 0x0, ttl 62, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 10.1.1.2: ICMP echo reply, id 24, seq 0, length 64

[eno16777728] 04:03:59.861185 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[eno16777728] 04:03:59.861946 IP (tos 0x0, ttl 63, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

OUT方向按时间排序
[veth2a3e623] 04:03:59.861136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.1.1.2 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[docker0] 04:03:59.861136 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.1.1.2 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[eno16777728] 04:03:59.861185 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

IN方向按时间排序
[eno16777728] 04:03:59.861946 IP (tos 0x0, ttl 63, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64
[docker0] 04:03:59.861979 IP (tos 0x0, ttl 62, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 10.1.1.2: ICMP echo reply, id 24, seq 0, length 64
[veth2a3e623] 04:03:59.861986 IP (tos 0x0, ttl 62, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 10.1.1.2: ICMP echo reply, id 24, seq 0, length 64

node102
==================
[veth8198030] 04:03:59.043575 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[veth8198030] 04:03:59.043621 IP (tos 0x0, ttl 64, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

[docker0] 04:03:59.043565 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[docker0] 04:03:59.043621 IP (tos 0x0, ttl 64, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

[eno16777728] 04:03:59.043509 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[eno16777728] 04:03:59.043634 IP (tos 0x0, ttl 63, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

IN方向按时间排序
[eno16777728] 04:03:59.043509 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[docker0] 04:03:59.043565 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64
[veth8198030] 04:03:59.043575 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 192.168.80.101 > 20.1.1.2: ICMP echo request, id 24, seq 0, length 64

OUT方向按时间排序
[veth8198030] 04:03:59.043621 IP (tos 0x0, ttl 64, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64
[docker0] 04:03:59.043621 IP (tos 0x0, ttl 64, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64
[eno16777728] 04:03:59.043634 IP (tos 0x0, ttl 63, id 32460, offset 0, flags [none], proto ICMP (1), length 84) 20.1.1.2 > 192.168.80.101: ICMP echo reply, id 24, seq 0, length 64

 

br0抓包无数据，gre通道是虚拟的，实际还是从物理网卡传输