(a super simple diagram to give you an idea of the network layout)
I’ve been working on my SimpleStack OpenStack install lately. This design currently uses a FlatDHCP nova-network.
I have a couple of hardware servers that I’m deploying it on at work, and I was having a ton of trouble with the public and flat interface being able to receive arp requests, but not sending them back.
It turns out the answer was simple: the vlan coming down the port to the interface was tagged.
This is part of the nova.conf file:
flat_network_bridge = br100
flat_interface = em2
public_interface = em2
Here’s some ARP traffic off the em2 interface.
curtis@compute1:~$ sudo tcpdump -n -e -ttt -i br100 arp
00:00:00.888602 5c:5e:ab:d9:e0:f0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 601, p 0, ethertype ARP, Request who-has x.y.z.22 tell x.y.z.18, length 42
00:00:00.900232 5c:5e:ab:d9:e0:f0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 601, p 0, ethertype ARP, Request who-has x.y.z.22 tell x.y.z.18, length 42
00:00:00.600620 5c:5e:ab:d9:e0:f0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 601, p 0, ethertype ARP, Request who-has x.y.z.22 tell x.y.z.18, length 42
00:00:00.703113 5c:5e:ab:d9:e0:f0 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 60: vlan 601, p 0, ethertype ARP, Request who-has x.y.z.22 tell x.y.z.18, length 42
As you can see, we have “ethertype 802.1Q” and vlan 601. It took me quite a while to notice that, but once I did everything made sense.
The arp requests were getting to em2, but because they were tagged with vlan 601 the server wouldn’t reply because it wasn’t setup to use that vlan.
As soon as I added the 601 vlan and configured nova to use it, everything started working.
curtis@compute1:~$ sudo modprobe 8021q
curtis@compute1:~$ sudo vconfig add em2 601
Added VLAN with VID == 601 to IF -:em2:-
curtis@compute1:~$ sudo ifconfig em2.601 up
The I changed the nova.conf file to look like this:
flat_network_bridge = br100
flat_interface = em2.601
public_interface = em2.601
Then I restarted nova-network.
Finally I ended up with this bridge config:
curtis@compute1:/etc/nova# sudo brctl show
bridge name bridge id STP enabled interfaces
br100 8000.782bcb617008 no em2.601
vnet0
virbr0 8000.000000000000 yes
And I was able to have my flat network, which is all public IPs, working.
Now, there’s no reason (in this situation) for this port to have the vlan tagged on it, we just need that one network on the physical interface, but it was tagged and in this case it was just easier to go with that then to try to have any changes made on the network operator side.
For a while we were chasing red herrings on this problem, and finally I just took a hard look at the arp packets and realized it was a tagged vlan. Obvious now, but it took a while to figure out just how obvious.
It was a good feeling to realize the problem, fix it, and to get SimpleStack up and running on physical computers with public IPv4 addresses so that I can continue to experiment with small OpenStack installations. I’ve already realized this network design probably wouldn’t work for anyone who doesn’t have a lot of IPv4 addresses or can’t support IPv6. IPv6 would be so great for this. When will we really start using it…ISPs I’m looking at you…