XenServer Network Bonding

I've just had a bit of a learning experience with respect to Citrix XenServer and what it refers to as network bonding, and thought I would share here.

All of what I am about to say comes with the disclaimer that I am not a network engineer, but rather a server engineer, so my networking terms may not be 100% accurate.

I've been working to deploy a new XenServer pool here at $WORK recently, and we've been working under the assumption that we would connect it to our new datacenter networking standard, which we refer to locally as a "virtual chassis" -- two in-rack switches "stacked" together connected independently via 10GE to two stacked aggregator switches and network traffic tagged with standard 802.1q vlan tags.  In order for this to work as a fault tolerant configuration, each server must have a connection to each of the in-rack switches.  On our normal, non-virtualized linux server deployments we have been using standard link aggregation (otherwise known as bonding or NIC teaming) with LACP (Link Aggregation Control Protocol).

According to the documentation, XenServer, which is really just Linux under the hood, supports network bonding, so this should be the same, right?

Wrong.

Rather than using LACP, XenServer actually uses something called "Source Level Balancing," which is based on the standard Linux "Adaptive Load Balancing" method of bonding (See Citrix's KB article CTX124421).  The really cool part of this bonding mode is that it requires no knowledge on the switch side of the connection to make it work.  Instead, the hypervisor migrates VM network traffic from port to port by sending a "gratuitous ARP" through the new port with the MAC address of the VM to be moved.  In an active-active configuration, XenServer monitors the traffic for each of the running VMs, and will rebalance as needed every few seconds (according to the manual, 10 seconds).

So, don't do what I did and configure the switch ports that XenServer uses as LAG groups.  Otherwise you'll end up with a bonding mismatch and waste time trying to figure out why you are having weird networking issues.