creating vlan over openstack (by quqi99)

Posted 2021-10-23 quqi99

版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 (作者：张华 发表于：2021-06-29)

问题

测试环境是openstack over openstack的，在underlying openstack中我的tenant之下本来只有一个名为zhhuabj_admin_net的vxlan网络, 然后upper openstack用这个vxlan网络提供ext_net的flat网络，upper openstack再提供一个名为private的vxlan网络。
那现在我想在upper openstack中测试vlan网络，另外，也没有将物理交换机改trunk的权限，那该如何改造呢？

回顾vlan

neutron中正常配置vlan的流程如下，但确保一点外部物理交换机一定要配置成trunk模式允许1001

juju config neutron-openvswitch bridge-mappings="physnet1:br-data"
juju config neutron-openvswitch data-port="br-data:ens9"
juju config neutron-api vlan-ranges="physnet1:1000:2000"
# must configure external switch to truck mode to allow vlan 1000 to 2000
neutron net-create net1 --provider:network_type vlan --provider:physical_network physnet1 --provider:segmentation_id 1001

在这种方式中，vlan是在交换机(br-int)中打的（对于neutron trunking则是在虚机内部打tag):

• 这种模式，虚机里不打vlan
• 到br-int后打上vlan tag (虚机与br-int的那个port叫access port, 对于access port, 进来和出去为untagged报文，进到交换机后再按照port上的tag配置进行vlan划分和处理)
• 到物理交换机后，由于有配置trunk, trunk port可收发带vlan报文并且出去后不剥离vlan tag
• 到另一虚机的br-int再去掉vlan tag即可

改成neutron trunking之后，将允许在虚机内部打tag:

• 虚机里现在能打tag，意味着将有eth0.1001, 此时从虚机出来带tag
• 虚机eth0之前在neutron中的port叫trunkport, 可以正常收发带vlan报文并且出去后的不剥离vlan
• 此时，带vlan的报文可从openstack原有的gre/vxlan/flat等出去，这样不需要改变沿路物理交换机的trunk设置
• 到另一虚机后再去tag

回顾Neutron trunking

这里面就要用到neutron trunking特性，这篇[文章]讲得挺好的。(https://blog.csdn.net/bc_vnetwork/article/details/53927687)[1]


使用了neutron trunking port之后的虚机里面就可以打vlan了，这样就可以同时在虚机里面通过一个网卡支持众多网络。具体使用步骤是：

openstack network create net0
openstack network create net1
openstack network create net2
openstack subnet create --network net0 --subnet-range 10.0.4.0/24 subnet0
openstack subnet create --network net1 --subnet-range 10.0.5.0/24 subnet1
openstack subnet create --network net2 --subnet-range 10.0.6.0/24 subnet2
openstack network trunk create --parent-port trunkparent \\
 --subport port=subport1,segmentation-type=vlan,segmentation-id=1 \\
 --subport port=subport2,segmentation-type=vlan,segmentation-id=2 mytrunk

# inside vm
sudo ip link add link eth0 eth0.1 address fa:16:3e:cc:b9:27 broadcast ff:ff:ff:ff:ff:ff type vlan id 1
sudo ip link add link eth0 eth0.2 address fa:16:3e:25:d2:c9 broadcast ff:ff:ff:ff:ff:ff type vlan id 2
sudo ip link set eth0.1 up
sudo ip link set eth0.2 up
sudo dhclient eth0.1
sudo dhclient eth0.2
ping -I eth0.1 10.0.5.2
tcpdump -en -i qvob7d4c968-af

改造原理

upper openstack中，定义neutron vlan 1000网络

-  ./bin/neutron-ext-net-ksv3 --project admin --network-type flat -g $GATEWAY -c $CIDR_EXT -f $FIP_RANGE ext_net
+  ./bin/neutron-ext-net-ksv3 --project admin --network-type vlan --vlan-id 1000 -g $GATEWAY -c $CIDR_EXT -f $FIP_RANGE ext_net

upper openstack的这个ext_net作为一个provider network使用的是underlying openstack的zhhuabj_admin_net vxlan网络。
这时候，不像普通的neutron trunking是在虚机里面打vlan，这时候由于upper openstack定义了vlan网络，upper openstack里的虚机里面是不用打vlan的(vlan还是虚机所有的br-int中再打)
这样，在underlying openstack中再定义一个名为zhhuabj_admin_net2的vxlan网络专用于trunk

source ~/novarc
openstack router create zhhuabj_router2
openstack network create --disable-port-security zhhuabj_admin_net2
openstack subnet create --subnet-range 10.10.0.0/24 --network zhhuabj_admin_net2 --allocation-pool start=10.10.0.50,end=10.10.0.100 --gateway 10.10.0.1 zhhuabj_admin_net2_subnet
openstack router add subnet zhhuabj_router2 zhhuabj_admin_net2
openstack router set --external-gateway zhhuabj_admin_net2 zhhuabj_router2

这样，可用下列命令为upper openstack的某一节点的port做trunk (上层的所有计算节点和neutron-gateway节点所用的port均要转换成trunk)

+            parent_port_id=$(openstack port create $current_model-data-port-$i --network lathiat_vlan_net --no-fixed-ip -f value -c id)
+            child_port_id=$(openstack port create $current_model-child-port-$i --network $network --no-fixed-ip -f value -c id)
+            openstack network trunk create $current_model-trunk0-$i --parent-port $parent_port_id --subport port=$child_port_id,segmentation-type=vlan,segmentation-id=1000 

这样，upper openstack使用vlan网络出来的流量会在br-int处打上vlan, 到它所在的计算节点和neutron-gateway的port在underlying openstack中又转成了trunk，这样underlying openstack会将上层来的vlan流量仍然以vxlan发出去，这样真正的物理环境是不需要修改物理交换成打trunk的。

实际步骤

1, underlying openstack中创建专用trunk的zhhuabj_admin_net2 (vxlan)

source ~/novarc
openstack router create zhhuabj_router2
openstack network create --disable-port-security zhhuabj_admin_net2
openstack subnet create --subnet-range 10.10.0.0/24 --network zhhuabj_admin_net2 --allocation-pool start=10.10.0.50,end=10.10.0.100 --gateway 10.10.0.1 zhhuabj_admin_net2_subnet
openstack router add subnet zhhuabj_router2 zhhuabj_admin_net2
openstack router set --external-gateway zhhuabj_admin_net2 zhhuabj_router2

2, 打上patch
3, 创建upper openstack

./generate-bundle.sh --defaults --name dt --create-model -r stein -s bionic --revision-info ./xxx.bundle --num-compute 2 --run

4, 将计算节点所用的port在底层openstack中转成trunk port

./configure

5, 将neutron-gateway节点所用的port在底层openstack中转成trunk port

# 由于之前neutron-gateway已经创建了data-port，须将它从底层openstack中将这个port删除，再运行下列命令
./bin/add-data-ports.sh neutron-gateway

6, 创建测试虚机

openstack keypair create --public-key ~/.ssh/id_rsa.pub mykey
openstack server create --wait --image bionic --flavor m1.small --key-name mykey --nic net-id=$(openstack network show ext_net -c id -f value) i1
openstack server create --wait --image bionic --flavor m1.small --key-name mykey --nic net-id=$(openstack network show ext_net -c id -f value) i2

7, 上步不work，打开dhcp后work

openstack subnet set --dhcp ext_net_subnet
nova reboot --hard i1
nova reboot --hard i2

8, 其他信息
即从zhhuabj_admin_net来的vlan流量转成TRUNK后从zhhuabj_admin_net2中通过vxlan发给其他物理机。

$ source ~/novarc
$ openstack network trunk list
+--------------------------------------+-------------+--------------------------------------+-------------+
| ID                                   | Name        | Parent Port                          | Description |
+--------------------------------------+-------------+--------------------------------------+-------------+
| 05814a30-1f72-4ca2-a2b8-c31de133fac5 | dt-trunk0-0 | c1aaa0ad-bd27-40c2-9c7f-76f415900b00 |             |
| e466bf99-1989-4f03-b179-e58e555591c0 | dt-trunk0-1 | e299e0d6-5037-40dc-b9d7-a539994de2af |             |
| ec6f6d96-960f-445e-80f3-a1695b07d954 | dt-trunk0-0 | 2abf732c-e3d3-4039-b43a-852bd703847c |             |
+--------------------------------------+-------------+--------------------------------------+-------------+
$ openstack network list |grep admin
| 6999cbde-1293-45a1-9c83-c10277885993 | zhhuabj_admin_net2       | ed151d07-574e-4d9f-8351-254f68af45a5 |
| b0268083-fcab-417b-b291-f1465880ee82 | zhhuabj_admin_net        | 3b593653-04fb-407a-9853-e7886a608cd7 |
$ openstack port show c1aaa0ad-bd27-40c2-9c7f-76f415900b00 |grep network_id
| network_id              | 6999cbde-1293-45a1-9c83-c10277885993
$ openstack network trunk show 05814a30-1f72-4ca2-a2b8-c31de133fac5
+-----------------+--------------------------------------------------------------------------------------------------+
| Field           | Value                                                                                            |
+-----------------+--------------------------------------------------------------------------------------------------+
| admin_state_up  | UP                                                                                               |
| created_at      | 2021-06-29T09:00:58Z                                                                             |
| description     |                                                                                                  |
| id              | 05814a30-1f72-4ca2-a2b8-c31de133fac5                                                             |
| name            | dt-trunk0-0                                                                                      |
| port_id         | c1aaa0ad-bd27-40c2-9c7f-76f415900b00                                                             |
| project_id      | 01c14ded0bf84e139c4f82316a377ca5                                                                 |
| revision_number | 2                                                                                                |
| status          | ACTIVE                                                                                           |
| sub_ports       | port_id='f281c260-d3ba-4481-b67b-83e73f095393', segmentation_id='1000', segmentation_type='vlan' |
| tags            | []                                                                                               |
| tenant_id       | 01c14ded0bf84e139c4f82316a377ca5                                                                 |
| updated_at      | 2021-06-29T09:01:24Z                                                                             |
+-----------------+--------------------------------------------------------------------------------------------------+
$ openstack port show f281c260-d3ba-4481-b67b-83e73f095393 |grep network_id
| network_id              | b0268083-fcab-417b-b291-f1465880ee82

$ source ~/stsstack-bundles/openstack/novarc
$ neutron net-list |grep ext_net
| 0bdab84d-0320-4796-be08-64c5d3ddbc11 | ext_net | 8201ba31-3dbb-41be-a118-944ccce62b6d 10.5.0.0/16     |

root@juju-3f7190-dt-7:~# ovs-appctl fdb/show br-int
 port  VLAN  MAC                Age
    1     1  1e:35:02:8a:23:f3  298
    1     1  fa:16:3e:b9:27:aa  283
    1     1  fa:16:3e:12:96:99  281
    1     1  3a:83:b2:d6:d9:f2  190
    4     1  fa:16:3e:d1:c2:86  116
    1     1  fa:16:3e:c9:07:83   24

Reference

[1] OpenStack Neutron新功能VLAN-aware-VMs介绍 - https://blog.csdn.net/bc_vnetwork/article/details/53927687