XenServer Management and Jumbo Frames

In a word, don't do it.

Perhaps some additional background would help. :)

We maintain many XenServer pools, most of which consist of four "compute" servers attached to a shared storage array.  Each server has two ethernets acting as a management network bond, as well as two ethernets acting as a bond for VM traffic.  The VM traffic is VLAN-tagged, the management traffic is not.

We had recently upgraded all of our pools to XenServer 6.1, a little faster than we typically would have so that we could gain access to some of the cool, new features (e.g., inter-pool VM migration).  Life is good, everything works fine.  Until it came time to apply a couple of patches.  After applying a patch I would reboot the server, at which point it would momentarily re-contact the pool and then disappear.  The Xapi services on the host would not respond, and the pool master would not acknowledge the node's presence.  SSH connectivity to the node worked, however.

This issue proved to be pre-existing, as in the patches were not what caused the problem.  I tried rebooting a node that had vanilla XS 6.1 and it exhibited similar symptoms.  It was just coincidental that the servers had not been rebooted until it came time to apply patches.

After some experimentation and trial and error, I was able to [reliably] get the node back online by performing an "emergency network reset" and rebooting.  However, the node would rejoin successfully only until the next reboot, whereupon it became a case of rinse, lather, repeat.

Further trial-and-error showed that if I removed the management bond entirely and ran all management traffic through a single interface, reboots worked properly and as expected (i.e., the system would seamlessly rejoin the pool).  Recreate the bond and the problem re-manifested.

Hrm.

After a period of tearing out my hair over this, I noticed the MTU setting.  We typically configure our VM traffic bonds with an MTU of 9000 so that customers can use so called "jumbo frames" within their VMs.  Without putting too much thought into it, we had also been configuring our management bonds with MTU=9000 as well.  On a hunch, I re-created the management bond, but this time with a default MTU of 1500.  Rebooted the node and....SUCCESS!  It correctly re-joined the pool after a reboot.

So, the moral of the story seems to be that if you have XenServer 6.1 installed on a system with a bonded management interface, ensure that bond has the default MTU of 1500.  Jumbo frames seem to make it unhappy for reasons unknown to me.  We've had these bonds enabled for quite some time -- this behavior seems to be new with version 6.1.  I haven't yet contacted Citrix to see if they are aware of the issue or not, but I thought I would at least document the issue here, in case someone else out there runs into similar problems.  I know that my many, many google searches on the matter ended up being fruitless.

The silver lining in this particular cloud is that throughout all this mess, all of our virtual machines stayed online and had no issues whatsoever, so our customers were never even aware there was a problem!  That has to count for something...

Linux Kernel and Leap Seconds

We had several systems at $WORK tonight become somewhat unresponsive.  CPU usage was pegged, and  interactive response times were typically abysmal, sometimes waiting several seconds and/or minutes for keystrokes to be acknowledged.

In addition, some systems were responding well enough, but had very high context switch rates.  The lowest rate I saw was about 500,000 context switches a second, but the highest I saw was over 2.8 million switches a second!

Almost all of the poorly-performing systems were virtual servers, and the performing-okay-but-high-context-switch-rate systems were all physical servers.  I hypothesized that the problem was the same one, but the physicals had more CPU power available to them -- most of our VMs don't have more than 2-3 vCPUs, whereas the physicals have upwards of 16 in some cases, if you count hyperthreads.

As it turns out, it was being caused by some weird (sorry, I'll try to keep from adding any more of this technical jargon...) kernel interaction when it processed the leap second that occurred today.  For more details, I turn you now to the blog entry that helped me narrow the problem down and provided me with a simple fix:

http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

For the record, should this link ever stop working, he said:

The fix is quite simple – simply set the date. Alternatively, you can restart the machine, which also works. Restarting MySQL (or Java, or whatever) does NOT fix the problem. We put the following into puppet to run on all our machines:

$ cat files/bin/leap-second.sh 
#!/bin/bash
# this is a quick-fix to the 6/30/12 leap second bug

if [ ! -f /tmp/leapsecond_2012_06_30 ]
then
/etc/init.d/ntpd stop; date -s "`date`" && /bin/touch /tmp/leapsecond_2012_06_30
fi

His solution was a lot more elegant than mine, which was to simply reboot the system. :)  It was also a lot easier to apply prophylactically to our entire fleet.

Piecaken for Pi Day

Awhile ago I ran across a flickr picture of a "piecaken."  For the uninitiated, a piecaken is a cake with one or more pies baked into it.  The version in the link is a two-layer variety, but I realized that a single layer wouldn't actually be too difficult.  I also made the mistake of mentioning it at the office shortly after the announcement of the annual pi day celebration where we bring in pies of all shape to share with each other in celebration of this well known mathematical constant.  Unfortunately, I mentioned it within earshot of my director, who also happened to be the organizer of this party, and she virtually insisted that I had to bring one.  Despite my trepidations, not only in my rather lacking skill-set in this area (I'm not exactly the world's greatest baker), but also whether or not it would taste any good, I set forth to create one.

Goodbye Cable TV...Almost

I've been working for the past few weeks to prepare for the severing of relationships with our current cable TV provider -- signing up for Hulu, building a HTPC (Home Theater Personal Computer, for the uninitiated), etc.  Last night, I completed the last few steps -- powered down the TiVo and removed the CableCARDs in it.  Then, this morning, I called to cancel my account.

Except I didn't do it.

As I had expected, I was passed over to the "retentions" department and I was given a few options to entice me to stay.  What I wasn't expecting, however, were the plans that didn't have any promotional pricing, and pretty reasonable prices at that.  I ended up scaling us back to the "bare bones minimum" plan that only gives us the network channels plus a smattering of basic cable channels (Discovery being the only one he listed that we watch with any frequency).  This plan is about $21/mo (probably $25 after taxes), which compared to the $98 we had been paying before (plus the $13/month going to TiVo, which is also going to get cancelled in favor of the HTPC), is still a pretty reasonable savings -- about $86 per month, or just over $1000 a year!

I'm not viewing this as "caving in" to their attempts to retain me as a paying customer, and part of me was expecting this to be the end outcome anyway, for the following reasons:

• While I have managed to locate online alternatives to almost all of our shows, the ones we like from Discovery do not really seem to be online anywhere, other than paying per episode from Amazon.  Plus, Discovery can be one of those great put-it-on-and-forget-about-it channels, for example when I'm sick on the couch.
• Part of my plan had included getting the network stations "OTA" (over the air) with an antenna.  Unfortunately, I don't yet have an antenna in place, nor do I have the structured cabling installed yet to get the signal from it to the TV.  Since this new cable plan has no commitment period, this can act, if nothing else, as a stop-gap until we can get the antenna set up.

So, we shall see how this works....

XenServer Network Bonding

I've just had a bit of a learning experience with respect to Citrix XenServer and what it refers to as network bonding, and thought I would share here.

All of what I am about to say comes with the disclaimer that I am not a network engineer, but rather a server engineer, so my networking terms may not be 100% accurate.

I've been working to deploy a new XenServer pool here at $WORK recently, and we've been working under the assumption that we would connect it to our new datacenter networking standard, which we refer to locally as a "virtual chassis" -- two in-rack switches "stacked" together connected independently via 10GE to two stacked aggregator switches and network traffic tagged with standard 802.1q vlan tags.  In order for this to work as a fault tolerant configuration, each server must have a connection to each of the in-rack switches.  On our normal, non-virtualized linux server deployments we have been using standard link aggregation (otherwise known as bonding or NIC teaming) with LACP (Link Aggregation Control Protocol).

According to the documentation, XenServer, which is really just Linux under the hood, supports network bonding, so this should be the same, right?

Wrong.

Rather than using LACP, XenServer actually uses something called "Source Level Balancing," which is based on the standard Linux "Adaptive Load Balancing" method of bonding (See Citrix's KB article CTX124421).  The really cool part of this bonding mode is that it requires no knowledge on the switch side of the connection to make it work.  Instead, the hypervisor migrates VM network traffic from port to port by sending a "gratuitous ARP" through the new port with the MAC address of the VM to be moved.  In an active-active configuration, XenServer monitors the traffic for each of the running VMs, and will rebalance as needed every few seconds (according to the manual, 10 seconds).

So, don't do what I did and configure the switch ports that XenServer uses as LAG groups.  Otherwise you'll end up with a bonding mismatch and waste time trying to figure out why you are having weird networking issues.

Fixing a Galaxy Tab 10.1 "Infinite Reboot" State

I've had this documented in my personal notes for awhile, but it dawned on me today that perhaps others out there may benefit from my experience, so I am going to post this publicly.

Last year Google provided attendees of Google I/O a very nice tablet -- the Samsung Galaxy Tab 10.1, not to be confused with the retail version of the 10.1, this edition has the "army of androids" on the back, and has a slightly different version of Android on it.  Unfortunately, it also seems to have a habit of occasionally getting stuck into an "infinite reboot" state.  The tablet will freeze, restart, run through the initial startup animation, pause, and then restart again.  Repeat ad nauseam until the battery dies.  I've had my tablet get into this state twice since I got it.  Rumor has it, if you report the problem to Samsung, it is a ship-back-to-the-manufacturer type of problem, but if you have a few technical skills, this may not be necessary.

Mobile blogging

Huh, look at that.  Mobile blogging.

Trying this again

It seems like every few months I get the idea that it might be handy to have a blog.  This is just the latest iteration.