We run several virtual machine host servers on a network with multiple VLANs. The virtual machines are members of different VLANs, but are not themselves aware of the VLAN. This is how we did it.
We use KVM for virtualisation, but I’m not going to discuss the details here. Like any virtualisation solution, KVM starts a virtual machine and attaches its virtual network hardware to a network interface on the host OS. What I want to discuss is how to implement the networking layer.
I previously wrote about creating a network bridge for Virtualbox virtual machines, and we’re going to do something similar here. However, we want to implement VLAN support and native 802.1q VLAN tagging at the same time.
The design:
- The physical server is connected to the Ethernet switch via a 802.1q VLAN tagged trunk port
- The host OS is aware of the trunk port, and implements several virtual network interfaces. Each virtual interface is associated with one VLAN. Any traffic on that virtual interface exits the physical network interface as tagged VLAN packets.
- The host OS provides a network bridge for each VLAN, and adds the virtual VLAN interface to the bridge.
- The guest OSes (virtual machines) are not VLAN-aware. They have a normal ethernet interface which requires no special configuration.
- The virtualisation software (KVM, in this case) attaches the network interface of the guest OS to the VLAN-specific network bridge.
Warning: changing the Ethernet setup to your server can cause you to lose access to it. At all times during this process, ensure you have console access to the server, and network or console access to the switch.
That said, we’ll try to time the network interruption such that we don’t lose access. First, configure your server for VLAN networking. This guide assumes the server runs Ubuntu 8.04 “Hardy Heron”, but the steps are similar for any recent Ubuntu release. I have no idea how Red Hat handles VLANs and bridging, but I invite you to provide the steps in comments.
This guide also assumes the server is connected to a VLAN unaware (access) port, on VLAN 100, with an existing /etc/network/interfaces like so:
# The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 192.168.0.2 netmask 255.255.255.0 gateway 192.168.0.1
Finally, we assume that virtual machines will be attached to either VLAN 100 (the same as the host OS’s own traffic) or VLAN 200. So we’ll prepare support for both.
Install vlan and bridge support.
apt-get install vlan bridge-utils
Then edit /etc/network/interfaces. Modify the existing network setup to be aware of VLAN 100, and to create a bridge on it.
# The loopback network interface auto lo iface lo inet loopback # LAN uses VLAN trunking, so set any IP addresses on appropriate bridge auto eth0 iface eth0 inet manual up ifconfig eth0 up # VLAN 100 auto eth0.100 iface eth0.100 inet manual up ifconfig eth0.100 up # KVM bridge, VLAN 100, via eth0 auto br100 iface br100 inet static address 192.168.0.2 netmask 255.255.255.0 gateway 192.168.0.1 bridge_ports eth0.100 bridge_maxwait 5 bridge_fd 1 bridge_stp on
We must bring eth0 up before we can create eth0.100, and we must bring eth0.100 up before we can attach it to the bridge interface. Finally, we must configure the host OS’s IP address on the bridge. If you don’t want to attach virtual machines to VLAN 100, you could configure the IP directly on eth0.100 and leave out the stanza for br100.
Now, apply changes. Because we’re going to move the existing IP from eth0 to br100, the easiest way is to reboot. Alternatively, you may try /etc/init.d/networking restart, but make sure you run this from console so you can fix any problems.
Network interruption begins as soon as you run the above command, or reboot. To restore access to the host OS, we must now configure the Ethernet port on the switch. For Cisco, this is easy. Assuming the server is connected to gigabit Ethernet port 1, run:
configure terminal interface GigabitEthernet0/1 switchport mode trunk
You should now be able to ping your server. Once you verify that works, add the configuration for VLAN 200 to /etc/network/interfaces.
# VLAN 200 auto eth0.200 iface eth0.200 inet manual up ifconfig eth0.200 up # KVM bridge, VLAN 200, via eth0 auto br200 iface br200 inet manual bridge_ports eth0.200 bridge_maxwait 5 bridge_fd 1 bridge_stp on
VLAN 200 doesn’t need an IP on the host OS, so it lacks the static IP configuration. Configure any additional VLANs the same way.
Now we are ready to attach virtual machines to the new bridges. For KVM, use virt-install to create a machine with 20 GB hard disk, 1 GB of RAM, booting the hardy iso, and attached to VLAN 100.
virt-install --connect qemu:///system -n guestname -r 1024 -f /path/to/virtual/disks/guestname.qcow2 -s 20 -c /path/to/isos/ubuntu-8.04.3-server-i386.iso --vnc --os-type linux --os-variant ubuntuHardy --accelerate --network=bridge:br100
Now use virt-manager to connect to the guest and configure it.
The last step can be replaced with any other virtualisation solution. For instance, with Virtualbox, you can create a VM and then attach it to the bridged interface br100.
Unlike the Virtualbox NAT/routing setup, we don’t need to enable IP forwarding (sysctl -w net.ipv4.ip_forward=1). This method uses layer 2 switching only. However, if you use an iptables firewall, make sure the FORWARD chain of the filter table defaults to “ACCEPT”, or use an equivalent rule matching traffic to/from the virtual bridges.
Tags: hardy, kvm, networking, ubuntu, virtualbox
-
Great page – just one question: I’m not getting the function of the “up ifconfig eth0.100 up” inside the very *same* interface’s “iface eth0.100 inet manual” stanza.
Doesn’t, at boot, an ifup -a scan for all “auto ” lines and issue an “ifconfig up” for them? (Indeed, isn’t debian’s reliance on ifconfig instead of iproute2 tools an oft-cited annoyance, but one that forces this behaviour in this instance?) Or is there some implication behind the definition of the stacked interfaces as “manual” that I’m missing?
Cheers!
Jonathan-
I have found that interfaces using the manual method don’t come up automatically. The commands that configure them, and bring them up, must be manually executed.
-
(sent to me by email, but pasted as comment so others can benefit)
I followed you post to configure my network but its not working.
My server is connected to switch over trunk line and I want to put server on vlan4 and in virtualbox all the vms in vlan5
host server with static ip in vlan4
I followed the steps from the post below it only works for vlan4 e.g. when I start my vm with br4 and manual vlan4 ip it communicates over the network but the problem is when I assign vlan5 ip with br5 it doesnt communicate over the network, no packets are received
Please help me understand what I am doing wrong?
Thank you very much in advance
-
Hi Amrit,
If you just want the host on VLAN4 and the guests all on VLAN5, you only need a bridge for VLAN5. Use something like this:
auto eth0.4 iface eth0.4 inet static address 192.168.0.2 netmask 255.255.255.0 gateway 192.168.0.1 auto eth0.5 iface eth0.5 inet manual up ifconfig eth0.5 up auto br5 iface br5 inet manual bridge_ports eth0.5 bridge_maxwait 5 bridge_fd 1 bridge_stp on
Then bring all VMs up with interfaces bridged to br5.
-
Hi Tyler, would it work your last answer without setting an IP to br5?
-
Hi Lukas,
Of course, I suggest you try. :)
Yes, it will work. Packets emitted by VMs on VLAN 5 will be tagged with VLAN 5 and emitted on eth0. You would want eth0 to be connected to an 802.1q-aware trunk port on a switch.
If you are sending tagged VLAN packets on eth0, you can bring up as many bridges as you want on different VLANs, and they don’t need IPs. They are bridges, a layer-2 technology, below IP. The problem you will then have is: how do the VMs on VLAN 5 route traffic? They can’t send it to your VM host as a default route. They’ll need a router upstream on VLAN 5.
-
Why create the br interfaces?
In virtualbox I am able to add the subinterface.. whats the difference? L3 vs L2?
-
It depends what you want to do. If you want your VMs accessible (networkable) from your host OS, but only accessing the Internet via NAT, the bridge + NAT method works well.
Recent versions of Virtualbox have added more control over NAT and internal networks, but I still prefer this method. To be fair, I haven’t tested these newer methods since this was written.
Virtualbox’s built-in methods are layer 2 as well. However, some don’t appear as devices to the host OS, and some do. Using a bridge allows me to sniff, firewall, and directly address all client OSes.
-
Here’s what I mean; I was kinda vague
So i’ve got two NICs.. I created a bond0 between the two, fed it a trunk link from the switch…
from there i’ve created the vlan subinterfaces bond0.200, bond0.4 etc..
I can add these directly into virtualbox with a random MAC… why is it important to go the extra step of creating the brX interfaces?
auto br5
iface br5 inet manual
bridge_ports eth0.5
bridge_maxwait 5
bridge_fd 1
bridge_stp onIts weird because it still works, but I feel like I am missing something critical
-
I should add that when I add it into virtualbox i have Bridged adapter selected.. how does that differ from creating the brX interface?
-
My goals are:
1. Allow the host to reach the client directly via networking. IE, SSH into the client OS.
2. Prevent the client OS from being reachable by outside devices. The NAT through the host does this, by acting as a default-deny firewall. This helps prevent the usual Windows virii from infecting VMs like an old XP instance I use for certain tasks.
3. Allow the client OS to reach outside devices. Again, the NAT through the host does this.
Directly attaching to a physical interface violates goal #2.
-
OK, now I have tested NAT, NatNetworks, and Internal with current VirtualBox.
NAT – Allows client to reach Internet via NAT
NatNetworks – Allows clients to reach Internet via NAT and to reach each other as if on a LAN
Internal – Allows clients to reach each other as if on a LANHowever, none of these allow the host OS to reach the client via ordinary networking. For this, you need a bridge.
-
-
-
-
-
-
-
13 comments
Comments feed for this article
Trackback link: https://www.tolaris.com/2010/02/20/vlans-bridges-and-virtual-machines/trackback/