1. OpenStack networking
- with Open vSwitch VLAN, GRE
Paul Sim
Cloud Consultant
paul.sim@canonical.com

2. Index
● Prior Knowledge
● OpenStack Networking - VLAN
● OpenStack Networking - GRE
● Security Group, Floating-IP, NameSpace
● Neutron ML2

3. Prior Knowledge - Network NameSpace
without Network NameSpace
Process
with Network NameSpace
Process
Process
Process
Process
Process
Process
Process
Share
Routing table
Ford
NameSpace
Benz
NameSpace
Network
Resources
Network Resources
BMW
NameSpace
Network
Resources
Network
Resources
Address
Netfilter rules
eth0
eth1
Network Resources
eth2
eth0
eth1
eth2
Network NameSpace provides isolation of the system resources associated with networking. Thus, each network
namespace has its own network devices, IP addresses, IP routing tables, /proc/net directory, port numbers, and so on.
- http://lwn.net/Articles/531114/

4. Prior Knowledge - VLAN, GRE
VLAN - Virtual LAN
802.1Q Header
TPIC : 16bit - 0x8100
TCI : 16bit
PCP : 3bit
DEI : 1bit
VID : 12bit (0 ~ 4095)
GRE - Generic Routing Encapsulation
16 Bytes Header + IP header
Key field : 32bit
- identify an individual traffic flow within a tunnel

5. OpenStack Installation - Grizzly
External network 192.168.122.0/24
eth0
eth0
Controller node
eth0
Network node
Quantum L3-agent
Nova
Keystone
Glance
Horizon
Quantum
openvswitch-agent
Quantum
metadata-agent
Quantum - Server
eth1
Compute node - 1
Compute node - 2
Quantum
openvswitch-agent
Quantum
openvswitch-agent
Nova compute
Nova compute
Quantum dhcpagent
eth1
eth0
eth2
eth2
eth1
eth2
Management 192.168.20.0/24
Data 192.168.10.0/24
eth1
eth2

6. Network Topology
●
●
●
●
ext_net : external network - 192.168.122.0/24
net_proj_one : “user_one” tenant - 50.50.1.0/24
net_proj_two : “user_one” tenant - 50.50.2.0/24
net_proj_new : “user_new” tenant - 60.60.1.0/24

7. Big picture - VLAN
OpenStack Grizzly OpenvSwitch plug-in VLAN mode
Network node
net_proj_one
net_proj_two
Compute node - 1
net_proj_new
VM
tap~
qr~
tap~
qr~
qg~
qg~
br-ex
qg~
VM
tap~
tag: 1
qr~
br-int
VM
tap~
tag:2
tap~
tag:2
tap~
int-br-eth1
phy-br-eth1
br-eth1
int-br-eth1
Data
192.168.10.0
/24
eth1
br-int
phy-br-eth1
eth1
br-eth1
eth0
OVS port
OVS Bridge
●
●
qg~~~ : external gateway interface
qr~~~ : virtual router interface

8. VLAN - Compute node
OpenStack Grizzly OpenvSwitch plug-in VLAN mode
Compute node - 1
br-eth1
eth1
VM
VM
VM
VM
tap~
tag: 1
tap~
tag:2
tap~
tag:2
tap~
tag:3
veth pair
phy-br-eth1
int-br-eth1
br-int
Packet conversion
mod_vlan_vid
mod_vlan_vid
Security Group[1]

9. VLAN - Compute node
Packet conversion
janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-eth1
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=90455.716s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=2 actions=drop
cookie=0x0, duration=89606.096s, table=0, n_packets=9484, n_bytes=2312018, priority=4,in_port=2,dl_vlan=1
actions=mod_vlan_vid:1024,NORMAL
cookie=0x0, duration=90456.248s, table=0, n_packets=6813, n_bytes=1325511, priority=1 actions=NORMAL
janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=90458.482s, table=0, n_packets=64, n_bytes=4644, priority=2,in_port=1 actions=drop
cookie=0x0, duration=89608.755s, table=0, n_packets=6499, n_bytes=1283680, priority=3,in_port=1,dl_vlan=1024
actions=mod_vlan_vid:1,NORMAL
cookie=0x0, duration=90459.075s, table=0, n_packets=9820, n_bytes=2323195, priority=1 actions=NORMAL
openvswitch-agent.log
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-int', 'hard_timeout=0,
idle_timeout=0,priority=3,in_port=1,dl_vlan=1024,actions=mod_vl
an_vid:1,normal']
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ovs-ofctl', 'add-flow', 'br-eth1', 'hard_timeout=0,
idle_timeout=0,priority=4,in_port=2,dl_vlan=1,actions=mod_vlan
_vid:1024,normal']

10. VLAN - Network node
OpenStack Grizzly OpenvSwitch plug-in VLAN mode
Network node
tap~
Namespcae
tap~
Namespcae
qr~
qg~
qr~
qg~
veth pair
br-int
int-br-eth1
phy-br-eth1
br-ex
eth0
Packet conversion
net_proj_one
mod_vlan_id
net_proj_two
Floating-IP(NAT)
net_proj_new
mod_vlan_id
eth1
qg~
Namespcae
br-eth1
qr~
tap~

11. VLAN - Network node
Packet conversion
janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-int
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=7370.307s, table=0, n_packets=6, n_bytes=468, priority=2,in_port=6 actions=drop
cookie=0x0, duration=7368.424s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=6,dl_vlan=2048 actions=mod_vlan_vid:2,NORMAL
cookie=0x0, duration=7367.991s, table=0, n_packets=764, n_bytes=191460, priority=3,in_port=6,dl_vlan=1024 actions=mod_vlan_vid:3,
NORMAL
cookie=0x0, duration=7369.073s, table=0, n_packets=0, n_bytes=0, priority=3,in_port=6,dl_vlan=500 actions=mod_vlan_vid:1,NORMAL
cookie=0x0, duration=7370.924s, table=0, n_packets=549, n_bytes=104066, priority=1 actions=NORMAL
janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-eth1
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=7373.826s, table=0, n_packets=14, n_bytes=1104, priority=2,in_port=2 actions=drop
cookie=0x0, duration=7372.725s, table=0, n_packets=13, n_bytes=922, priority=4,in_port=2,dl_vlan=1 actions=mod_vlan_vid:500,NORMAL
cookie=0x0, duration=7371.663s, table=0, n_packets=519, n_bytes=103966, priority=4,in_port=2,dl_vlan=3 actions=mod_vlan_vid:1024,
NORMAL
cookie=0x0, duration=7372.09s, table=0, n_packets=9, n_bytes=634, priority=4,in_port=2,dl_vlan=2 actions=mod_vlan_vid:2048,NORMAL
cookie=0x0, duration=7374.384s, table=0, n_packets=764, n_bytes=191460, priority=1 actions=NORMAL

12. Big picture - GRE
OpenStack Grizzly OpenvSwitch plug-in GRE tunneling
Network node
qr~
qr~
VM
Tunnel
gre~
qg~
patch
patch
br-int
qg~
Data
192.168.10.0
/24
tap~
br-tun
qr~
tap~
qg~
VM
tap~
tag: 1
patch
tap~
net_proj_new
br-tun
net_proj_two
gre~
net_proj_one
Compute node - 1
tap~
tag:2
patch
br-int
br-ex
eth0
OVS port
OVS Bridge
●
●
qg~~~ : external gateway interface
qr~~~ : virtual router interface

13. GRE - Compute node
OpenStack Grizzly OpenvSwitch plug-in GRE tunneling
Compute node - 1
patch
VM
VM
VM
tap~
tag: 1
br-tun
gre~
VM
Tunnel
tap~
tag:2
tap~
tag:2
tap~
tag:3
patch
br-int
Packet conversion
mod_vlan_vid
set_tunnel id
Security Group[1]

14. GRE - Compute node
Packet conversion
janghoon@compute-1:~$ sudo ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=87770.027s, table=0, n_packets=0, n_bytes=0, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:
00 actions=mod_vlan_vid:1,output:1
cookie=0x0, duration=87770.09s, table=0, n_packets=8786, n_bytes=1893724, priority=4,in_port=1,dl_vlan=1
actions=set_tunnel:0x1,NORMAL
cookie=0x0, duration=87769.693s, table=0, n_packets=3031, n_bytes=617650, priority=3,tun_id=0x1,dl_dst=fa:16:3e:db:08:63
actions=mod_vlan_vid:1,NORMAL
cookie=0x0, duration=87769.966s, table=0, n_packets=6320, n_bytes=4432680, priority=3,tun_id=0x1,dl_dst=fa:16:3e:e0:73:95
actions=mod_vlan_vid:1,NORMAL
cookie=0x0, duration=87771.753s, table=0, n_packets=2921, n_bytes=951454, priority=1 actions=drop

15. GRE - Network node
OpenStack Grizzly OpenvSwitch plug-in GRE tunneling
Network node
tap~
Namespcae
tap~
Namespcae
qr~
Namespcae
qr~
qg~
patch
patch
br-int
br-ex
eth0
Packet conversion
net_proj_one
set_tunnel id
net_proj_two
Floating-IP(NAT)
net_proj_new
mod_vlan_id
Tunnel
gre~
qg~
qr~
br-tun
qg~
tap~

16. GRE - Network node
Packet conversion
janghoon@Network-node:~$ sudo ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=474674.446s, table=0, n_packets=7899, n_bytes=2572502, priority=3,tun_id=0x3,dl_dst=01:00:00:00:00:00/01:00:00:00:00:
00 actions=mod_vlan_vid:2,output:1
cookie=0x0, duration=473163.123s, table=0, n_packets=7876, n_bytes=2565284, priority=3,tun_id=0x4,dl_dst=01:00:00:00:00:00/01:00:00:00:00:
00 actions=mod_vlan_vid:3,output:1
cookie=0x0, duration=633937.826s, table=0, n_packets=10543, n_bytes=3426814, priority=3,tun_id=0x1,dl_dst=01:00:00:00:00:00/01:00:00:00:00:
00 actions=mod_vlan_vid:1,output:1
cookie=0x0, duration=473163.329s, table=0, n_packets=16484, n_bytes=3348666, priority=4,in_port=1,dl_vlan=3 actions=set_tunnel:0x4,
NORMAL
cookie=0x0, duration=474674.541s, table=0, n_packets=16864, n_bytes=3389132, priority=4,in_port=1,dl_vlan=2 actions=set_tunnel:0x3,
NORMAL
cookie=0x0, duration=633937.905s, table=0, n_packets=62044, n_bytes=37320316, priority=4,in_port=1,dl_vlan=1 actions=set_tunnel:0x1,
NORMAL
cookie=0x0, duration=472911.069s, table=0, n_packets=16335, n_bytes=3551350, priority=3,tun_id=0x4,dl_dst=fa:16:3e:89:fd:ce
actions=mod_vlan_vid:3,NORMAL
cookie=0x0, duration=474336.184s, table=0, n_packets=16360, n_bytes=3560332, priority=3,tun_id=0x3,dl_dst=fa:16:3e:d8:d5:29
actions=mod_vlan_vid:2,NORMAL
cookie=0x0, duration=474674.351s, table=0, n_packets=525, n_bytes=52427, priority=3,tun_id=0x3,dl_dst=fa:16:3e:69:ca:97
actions=mod_vlan_vid:2,NORMAL
cookie=0x0, duration=473162.912s, table=0, n_packets=197, n_bytes=19365, priority=3,tun_id=0x4,dl_dst=fa:16:3e:d6:b8:07
actions=mod_vlan_vid:3,NORMAL
cookie=0x0, duration=633937.746s, table=0, n_packets=6207, n_bytes=630043, priority=3,tun_id=0x1,dl_dst=fa:16:3e:c7:ec:bd
actions=mod_vlan_vid:1,NORMAL
cookie=0x0, duration=474794.912s, table=0, n_packets=36912, n_bytes=7440964, priority=3,tun_id=0x1,dl_dst=fa:16:3e:8b:a6:d7
actions=mod_vlan_vid:1,NORMAL
cookie=0x0, duration=636252.069s, table=0, n_packets=163, n_bytes=36046, priority=1 actions=drop

17. Security Group - VLAN, GRE
FORWARD
quantum-filter-top
quantum-openvswi-local
Security group is applied here
quantum-openvswi-FORWARD
quantum-openvswi-sg-chain
quantum-openvswi-iTAP_NUMBER
quantum-openvswi-sg-fallback
quantum-openvswi-oTAP_NUMBER
quantum-openvswi-sg-fallback

18. Security Group - VLAN, GRE
Chain quantum-openvswi-sg-chain (4 references)
target prot opt source
destination
quantum-openvswi-i21767f1f-4 all -- 0.0.0.0/0
0.0.0.0/0
quantum-openvswi-o21767f1f-4 all -- 0.0.0.0/0
0.0.0.0/0
quantum-openvswi-i7903fd30-7 all -- 0.0.0.0/0
0.0.0.0/0
quantum-openvswi-o7903fd30-7 all -- 0.0.0.0/0
0.0.0.0/0
ACCEPT all -- 0.0.0.0/0
0.0.0.0/0
PHYSDEV match --physdev-out tap21767f1f-45 --physdev-is-bridged
PHYSDEV match --physdev-in tap21767f1f-45 --physdev-is-bridged
PHYSDEV match --physdev-out tap7903fd30-74 --physdev-is-bridged
PHYSDEV match --physdev-in tap7903fd30-74 --physdev-is-bridged
Chain quantum-openvswi-i7903fd30-7 (1 references)
target prot opt source
destination
DROP
all -- 0.0.0.0/0
0.0.0.0/0
state INVALID
RETURN all -- 0.0.0.0/0
0.0.0.0/0
state RELATED,ESTABLISHED
RETURN icmp -- 0.0.0.0/0
0.0.0.0/0
RETURN tcp -- 0.0.0.0/0
0.0.0.0/0
tcp dpt:22
RETURN udp -- 50.50.1.3
0.0.0.0/0
udp spt:67 dpt:68
quantum-openvswi-sg-fallback all -- 0.0.0.0/0
0.0.0.0/0
Chain quantum-openvswi-o7903fd30-7 (2 references)
target prot opt source
destination
DROP
all -- 0.0.0.0/0
0.0.0.0/0
MAC ! FA:16:3E:DB:08:63
RETURN udp -- 0.0.0.0/0
0.0.0.0/0
udp spt:68 dpt:67
DROP
all -- !50.50.1.2
0.0.0.0/0
DROP
udp -- 0.0.0.0/0
0.0.0.0/0
udp spt:67 dpt:68
DROP
all -- 0.0.0.0/0
0.0.0.0/0
state INVALID
RETURN all -- 0.0.0.0/0
0.0.0.0/0
state RELATED,ESTABLISHED
RETURN all -- 0.0.0.0/0
0.0.0.0/0
quantum-openvswi-sg-fallback all -- 0.0.0.0/0
0.0.0.0/0
[1] Note, OpenStack uses iptables rules on the TAP devices such as “tap~~” to implement security groups,.
However, Open vSwitch is not compatible with iptables rules that are applied directly on TAP devices that are connected to an
Open vSwitch port.

19. Network NameSpace
janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 ifconfig
lo
Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
qg-fa243f49-d6 Link encap:Ethernet HWaddr fa:16:3e:9f:4b:63
inet addr:192.168.122.50 Bcast:192.168.122.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe9f:4b63/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
qr-bc654dc2-f1 Link encap:Ethernet HWaddr fa:16:3e:c7:ec:bd
inet addr:50.50.1.1 Bcast:50.50.1.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fec7:ecbd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 route
Kernel IP routing table
Destination Gateway
Genmask
Flags Metric Ref Use Iface
default
192.168.122.1 0.0.0.0
UG 0 0
0 qg-fa243f49-d6
50.50.1.0
*
255.255.255.0 U 0 0
0 qr-bc654dc2-f1
192.168.122.0 *
255.255.255.0 U 0 0
0 qg-fa243f49-d6

20. Floating-IP(NAT) - VLAN, GRE
NameSpace
janghoon@Network-node:~$ sudo ip netns show
qdhcp-4c2f2346-ffaa-41a0-ab76-34cadf0163f5
qrouter-e1b88ce4-51e9-4744-be80-d70d04c6a59b
qdhcp-c19e22a0-1700-4b3b-91e5-2c961ef0a353
qrouter-244fff3f-f935-4bdd-949d-739f1ce81dd0
qdhcp-f37b681a-4be8-47b8-8063-3d17d24ee1ae
qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0
Floating-IP(NAT)
janghoon@Network-node:~$ sudo ip netns exec qrouter-cf5fe7b7-8fab-45de-ab1c-c0cd404ebed0 iptables -L -n -t nat
Chain quantum-l3-agent-PREROUTING (1 references)
target prot opt source
destination
REDIRECT tcp -- 0.0.0.0/0
169.254.169.254 tcp dpt:80 redir ports 9697
DNAT
all -- 0.0.0.0/0
192.168.122.51
to:50.50.1.2
Chain quantum-l3-agent-float-snat (1 references)
target prot opt source
destination
SNAT
all -- 50.50.1.2
0.0.0.0/0
to:192.168.122.51
Chain quantum-l3-agent-snat (1 references)
target prot opt source
destination
quantum-l3-agent-float-snat all -- 0.0.0.0/0
SNAT
all -- 50.50.1.0/24
0.0.0.0/0
0.0.0.0/0
to:192.168.122.50

21. Neutron ML2
The Modular Layer 2 (ML2) plugin is a framework allowing OpenStack Networking to simultaneously utilize the variety
of layer 2 networking technologies found in complex real-world data centers. It currently works with the existing
openvswitch, linuxbridge, and hyperv L2 agents, and is intended to replace and deprecate the monolithic plugins
associated with those L2 agents.
Neutron
ML2 Plugin
TypeDriver
Cisco Nexus
Arista
Flat
OpenDaylight
VxLAN
Hyper-V
GRE
OpenvSwitch
VLAN
MechanismDriver
pSwitch
TypeDriver : TypeDrivers maintain any needed type-specific network state, and perform provider network validation
and tenant network allocation.
MechanismDriver : The MechanismDriver is responsible for taking the information established by the TypeDriver and
ensuring that it is properly applied given the specific networking mechanisms that have been enabled.
https://wiki.openstack.org/wiki/Neutron/ML2

22. Neutron ML2
eth0
eth0
eth0
Network node
Compute node - 1
Compute node - 2
Neutron
ML2-agent
Neutron
ML2-agent
Nova compute
Nova compute
Neutron L3-agent
Neutron
ML2 plugin
Neutron metadataagent
Neutron dhcpagent
eth1
eth2
eth1
eth2
eth1
eth2