XenServer Management and Jumbo Frames

In a word, don't do it.

Perhaps some additional background would help. :)

We maintain many XenServer pools, most of which consist of four "compute" servers attached to a shared storage array.  Each server has two ethernets acting as a management network bond, as well as two ethernets acting as a bond for VM traffic.  The VM traffic is VLAN-tagged, the management traffic is not.

We had recently upgraded all of our pools to XenServer 6.1, a little faster than we typically would have so that we could gain access to some of the cool, new features (e.g., inter-pool VM migration).  Life is good, everything works fine.  Until it came time to apply a couple of patches.  After applying a patch I would reboot the server, at which point it would momentarily re-contact the pool and then disappear.  The Xapi services on the host would not respond, and the pool master would not acknowledge the node's presence.  SSH connectivity to the node worked, however.

This issue proved to be pre-existing, as in the patches were not what caused the problem.  I tried rebooting a node that had vanilla XS 6.1 and it exhibited similar symptoms.  It was just coincidental that the servers had not been rebooted until it came time to apply patches.

After some experimentation and trial and error, I was able to [reliably] get the node back online by performing an "emergency network reset" and rebooting.  However, the node would rejoin successfully only until the next reboot, whereupon it became a case of rinse, lather, repeat.

Further trial-and-error showed that if I removed the management bond entirely and ran all management traffic through a single interface, reboots worked properly and as expected (i.e., the system would seamlessly rejoin the pool).  Recreate the bond and the problem re-manifested.

Hrm.

After a period of tearing out my hair over this, I noticed the MTU setting.  We typically configure our VM traffic bonds with an MTU of 9000 so that customers can use so called "jumbo frames" within their VMs.  Without putting too much thought into it, we had also been configuring our management bonds with MTU=9000 as well.  On a hunch, I re-created the management bond, but this time with a default MTU of 1500.  Rebooted the node and....SUCCESS!  It correctly re-joined the pool after a reboot.

So, the moral of the story seems to be that if you have XenServer 6.1 installed on a system with a bonded management interface, ensure that bond has the default MTU of 1500.  Jumbo frames seem to make it unhappy for reasons unknown to me.  We've had these bonds enabled for quite some time -- this behavior seems to be new with version 6.1.  I haven't yet contacted Citrix to see if they are aware of the issue or not, but I thought I would at least document the issue here, in case someone else out there runs into similar problems.  I know that my many, many google searches on the matter ended up being fruitless.

The silver lining in this particular cloud is that throughout all this mess, all of our virtual machines stayed online and had no issues whatsoever, so our customers were never even aware there was a problem!  That has to count for something...

XenServer Network Bonding

I've just had a bit of a learning experience with respect to Citrix XenServer and what it refers to as network bonding, and thought I would share here.

All of what I am about to say comes with the disclaimer that I am not a network engineer, but rather a server engineer, so my networking terms may not be 100% accurate.

I've been working to deploy a new XenServer pool here at $WORK recently, and we've been working under the assumption that we would connect it to our new datacenter networking standard, which we refer to locally as a "virtual chassis" -- two in-rack switches "stacked" together connected independently via 10GE to two stacked aggregator switches and network traffic tagged with standard 802.1q vlan tags.  In order for this to work as a fault tolerant configuration, each server must have a connection to each of the in-rack switches.  On our normal, non-virtualized linux server deployments we have been using standard link aggregation (otherwise known as bonding or NIC teaming) with LACP (Link Aggregation Control Protocol).

According to the documentation, XenServer, which is really just Linux under the hood, supports network bonding, so this should be the same, right?

Wrong.

Rather than using LACP, XenServer actually uses something called "Source Level Balancing," which is based on the standard Linux "Adaptive Load Balancing" method of bonding (See Citrix's KB article CTX124421).  The really cool part of this bonding mode is that it requires no knowledge on the switch side of the connection to make it work.  Instead, the hypervisor migrates VM network traffic from port to port by sending a "gratuitous ARP" through the new port with the MAC address of the VM to be moved.  In an active-active configuration, XenServer monitors the traffic for each of the running VMs, and will rebalance as needed every few seconds (according to the manual, 10 seconds).

So, don't do what I did and configure the switch ports that XenServer uses as LAG groups.  Otherwise you'll end up with a bonding mismatch and waste time trying to figure out why you are having weird networking issues.