I was finally going to install RHEV on my brand-new system running RHEL 7 RC with KVM. However, it has a dependency on DNS. Which was fine, but my DNS server was on another network, not the private network used by KVM with the standard virtual bridge. To fix this, I chose to move my KVM installation to use Open vSwitch. I have written before about adding Open vSwitch to KVM as well as hooking VMs to the Open vSwitch; however, Open vSwitch 2.1 has its own idiosyncrasies, as does libvirt 1.1.
So, here are the steps to add Open vSwitch 2.x into RHEL 7. While I am using RHEL 7 Release Candidate, this also should apply to RHEL 7 GA.
Step 1: Get the Open vSwitch 2.x source and place on a build VM specific to RHEL 7
The best place to get the Open vSwitch source code is http://openvswitch.org. This I transferred to a virtual machine running on top of RHEL 7 that is designed purely to upgrade and build drivers for RHEL 7. We can call this VM “rhel7-build.” I would rather not have the entire build environment on my virtualization host. Actually, we do not even have a normal display head for this particular host.
Step 2: Build the Open vSwitch package for RHEL
An interesting aspect of Open vSwitch is that the RHEL 7 RC kernel already contains the openvswitch drivers, so all we need to do is build the management packages.# tar –xzf openvswitch-1.7.1.tar.gz # rpmbuild –bb rhel/openvswitch.spec
Transfer the resultant openvswitch packages to your RHEL7 KVM box and install them.
Step 3: Start Open vSwitch and verify that it is working
We need to set up the Open vSwitch and ensure that it is working and configured. Note that at this time, Open vSwitch and the standard virtual bridge for KVM are available for use. Actually, I even have four VMs running as well.# echo "BRCOMPAT=yes" >> /etc/sysconfig/openvswitch
Then, start the openvswitch service(s) and configure them to start on reboot as well:# /sbin/service openvswitch start # /sbin/chkconfig openvswitch on
Now, verify a few things:# virsh version Compiled against library: libvirt 1.1.1 Using library: libvirt 1.1.1 Using API: QEMU 1.1.1 Running hypervisor: QEMU 1.5.3 # lsmod |grep openvswitch openvswitch 70743 0 vxlan 37584 1 openvswitch gre 13808 1 openvswitch libcrc32c 12644 2 xfs,openvswitch# ovs-vsctl show XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ovs_version: "2.1.2"
All of this tells us we are ready to proceed.
Step 4: Create a few Open vSwitch bridges for management and virtual machine workloads
For security reasons, you will need two bridges or two VLANs: one for management traffic and one for non-management workloads. This is the best practice for a secure virtual environment. These steps have not changed from the first time I did this.
# ovs-vsctl add-br ovsbr0 # ovs-vsctl add-br ovsbr1
The next step differs depending on whether you are bonding physical NICs together or using a single NIC.
# ovs-vsctl add-bond ovsbr0 bond0 eth0 eth2 lacp=active # only needed for bonding # ovs-vsctl add-port ovsbr0 eth0 # only needed for single NIC installations # ovs-vsctl add-bond ovsbr1 bond1 eth1 eth3 lacp=active # only needed for bonding # ovs-vsctl add-port ovsbr1 eth1 # only needed for single NIC installations # ovs-vsctl add-port ovsbr0 mgmt0 -- set interface mgmt0 type=internal
However, this is far from complete. We need to modify the configuration files within /etc/sysconfig/network-scripts to make all the networks come back on reboot. So, the config scripts look like the following, depending on whether we are using bonding or not. First, we have to specify the bridges themselves as an OVSBridge type.
|# cat ifcfg-ovsbr0
|# cat ifcfg-ovsbr1
Now, define our local management port to use and ensure all networks come up in the proper form. For management, we define it as type OVSPort and specify that it is part of the OVS_BRIDGE named ovsbr0. We give it the IP address assigned to the machine, as well as the the proper netmask.
# cat ifcfg-mgmt0 DEVICE=mgmt0 ONBOOT=yes DEVICETYPE=ovs TYPE=OVSIntPort OVS_BRIDGE=ovsbr0 USERCTL=no BOOTPROTO=none HOTPLUG=no IPADDR=A.B.C.D NETMASK=255.255.255.192 GATEWAY=W.X.Y.Z DNS1=P.Q.R.S
Then we have to tell the network subsystem how to bring up each bridge’s physical component.
For a single NIC and the NIC bond (repeat for all bonds or single NICs in use):
|# cat ifcfg-eth0
NETBOOT=yes IPV6INIT=no BOOTPROTO=none NAME=”eth0″ DEVICETYPE=ovs TYPE=OVSPort OVS_BRIDGE=ovsbr0
|# cat ifcfg-bond0
For NIC bonds only, we tell the network subsystem how to bring up each physical interface (repeat for all NIC bonds in use):
|# cat ifcfg-eth0
|# cat ifcfg-eth1
The last thing we need to do is set the default route to use our new mgmt0 network for the KVM host. This should already happen for us, given the configurations, but for some reason it does not.
# cat mgmt0-route default via W.X.Y.Z
Now we are ready to restart our network subsystem. If you do this while VMs are running, they will lose network connectivity for a bit until you can fix the networks in use by each VM. This will disable all old virtual bridge mechanisms, as we are disabling the ebtables service. You can run both together, but for my lab, I prefer just to use Open vSwitch.
# service network restart # service ebtables stop # chkconfig ebtables off # ovs-vstl show XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX Bridge "ovsbr0" Port "ovsbr0" Interface "ovsbr0" type: internal Port "mgmt0" Interface "mgmt0" type: internal Port "eth0" Interface "eth0" ovs_version: "2.1.2" # ping -c 3 W.X.Y.Z PING W.Y.X.Z (W.Y.X.Z) 56(84) bytes of data. 64 bytes from W.Y.X.Z: icmp_seq=1 ttl=64 time=0.553 ms 64 bytes from W.Y.X.Z: icmp_seq=2 ttl=64 time=0.316 ms 64 bytes from W.X.Y.Z: icmp_seq=3 ttl=64 time=0.350 ms --- W.X.Y.Z ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.316/0.406/0.553/0.105 ms
Step 5: Define the bridge networks within virsh for use by our VMs
Until we define our network within virsh for use by our VMs, our new Open vSwitch will not be accessible for use or assignment. In the following, I created ovsbr0 and then ovsbr1 using the exact same method, just substituting the different name within the XML and commands.# cat ovsbr0.xml <network> <name>ovsbr1</name> <forward mode='bridge'/> <bridge name='ovsbr0'/> <virtualport type='openvswitch'/> <portgroup name='vlan-01' default='yes'> </portgroup> </network> # virsh net-define ovsbr1.xml # virsh net-list –all Name State Autostart Persistent -------------------------------------------------- default active yes yes ovsbr0 inactive no yes # virsh net-destroy default # virsh net-start ovsbr0 # virsh net-info ovsbr0 Name ovsbr0 UUID XXXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX Active: yes Persistent: yes Autostart: yes Bridge: ovsbr0 # virsh auto-start ovsbr0
Once completed, I ended up with the following:# virsh net-list --all Name State Autostart Persistent -------------------------------------------------- ovsbr0 active yes yes ovsbr1 active yes yes
As you can see, the bridges are available for use, and the old bridge is no longer defined.
Step 6: Assign the VMs
The penultimate step is to assign the VMs to the appropriate bridge. There are a number of ways to do this. If the VM is running, you can use virt-manager and graphically change the networking (which is what I did once my management network came back up, as my VMs were still running). Or you can shut down each VM using virsh and edit the XML by hand. If you go that route, you will change the networking segments to look like the following, which differs from the element used within Open vSwitch v1.# virsh edit vmname ... <interface type='network'> <mac address='52:54:00:XX:YY:ZZ'/> <source network='ovsbr1'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </interface> ...
Once you are done, your Open vSwitch definition may look like:# ovs-vsctl show XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX Bridge "ovsbr0" Port "ovsbr0" Interface "ovsbr0" type: internal Port "mgmt0" Interface "mgmt0" type: internal Port "eth0" Interface "eth0" Bridge "ovsbr1" Port "ovsbr1" Interface "ovsbr1" type: internal Port "vnet3" Interface "vnet3" Port "vnet4" Interface "vnet4" Port "vnet2" Interface "vnet2" Port "vnet0" Interface "vnet0" Port "vnet1" Interface "vnet1" Port "eth1" Interface "eth1" ovs_version: "2.1.2"
Now I have successfully migrated my KVM node to Open vSwitch 2.x, with all my VMs on the proper networks: one for management and one for workloads. The only next thing is to install RHEV-M within a VM, so that I can begin to manage the node. Since I am now using the proper networking, my standard DNS configuration now applies.