Linux Kernel and Leap Seconds

We had several systems at $WORK tonight become somewhat unresponsive.  CPU usage was pegged, and  interactive response times were typically abysmal, sometimes waiting several seconds and/or minutes for keystrokes to be acknowledged.

In addition, some systems were responding well enough, but had very high context switch rates.  The lowest rate I saw was about 500,000 context switches a second, but the highest I saw was over 2.8 million switches a second!

Almost all of the poorly-performing systems were virtual servers, and the performing-okay-but-high-context-switch-rate systems were all physical servers.  I hypothesized that the problem was the same one, but the physicals had more CPU power available to them -- most of our VMs don't have more than 2-3 vCPUs, whereas the physicals have upwards of 16 in some cases, if you count hyperthreads.

As it turns out, it was being caused by some weird (sorry, I'll try to keep from adding any more of this technical jargon...) kernel interaction when it processed the leap second that occurred today.  For more details, I turn you now to the blog entry that helped me narrow the problem down and provided me with a simple fix:

http://blog.mozilla.org/it/2012/06/30/mysql-and-the-leap-second-high-cpu-and-the-fix/

For the record, should this link ever stop working, he said:

The fix is quite simple – simply set the date. Alternatively, you can restart the machine, which also works. Restarting MySQL (or Java, or whatever) does NOT fix the problem. We put the following into puppet to run on all our machines:

$ cat files/bin/leap-second.sh 
#!/bin/bash
# this is a quick-fix to the 6/30/12 leap second bug

if [ ! -f /tmp/leapsecond_2012_06_30 ]
then
/etc/init.d/ntpd stop; date -s "`date`" && /bin/touch /tmp/leapsecond_2012_06_30
fi

His solution was a lot more elegant than mine, which was to simply reboot the system. :)  It was also a lot easier to apply prophylactically to our entire fleet.

Piecaken for Pi Day

Awhile ago I ran across a flickr picture of a "piecaken."  For the uninitiated, a piecaken is a cake with one or more pies baked into it.  The version in the link is a two-layer variety, but I realized that a single layer wouldn't actually be too difficult.  I also made the mistake of mentioning it at the office shortly after the announcement of the annual pi day celebration where we bring in pies of all shape to share with each other in celebration of this well known mathematical constant.  Unfortunately, I mentioned it within earshot of my director, who also happened to be the organizer of this party, and she virtually insisted that I had to bring one.  Despite my trepidations, not only in my rather lacking skill-set in this area (I'm not exactly the world's greatest baker), but also whether or not it would taste any good, I set forth to create one.

Goodbye Cable TV...Almost

I've been working for the past few weeks to prepare for the severing of relationships with our current cable TV provider -- signing up for Hulu, building a HTPC (Home Theater Personal Computer, for the uninitiated), etc.  Last night, I completed the last few steps -- powered down the TiVo and removed the CableCARDs in it.  Then, this morning, I called to cancel my account.

Except I didn't do it.

As I had expected, I was passed over to the "retentions" department and I was given a few options to entice me to stay.  What I wasn't expecting, however, were the plans that didn't have any promotional pricing, and pretty reasonable prices at that.  I ended up scaling us back to the "bare bones minimum" plan that only gives us the network channels plus a smattering of basic cable channels (Discovery being the only one he listed that we watch with any frequency).  This plan is about $21/mo (probably $25 after taxes), which compared to the $98 we had been paying before (plus the $13/month going to TiVo, which is also going to get cancelled in favor of the HTPC), is still a pretty reasonable savings -- about $86 per month, or just over $1000 a year!

I'm not viewing this as "caving in" to their attempts to retain me as a paying customer, and part of me was expecting this to be the end outcome anyway, for the following reasons:

• While I have managed to locate online alternatives to almost all of our shows, the ones we like from Discovery do not really seem to be online anywhere, other than paying per episode from Amazon.  Plus, Discovery can be one of those great put-it-on-and-forget-about-it channels, for example when I'm sick on the couch.
• Part of my plan had included getting the network stations "OTA" (over the air) with an antenna.  Unfortunately, I don't yet have an antenna in place, nor do I have the structured cabling installed yet to get the signal from it to the TV.  Since this new cable plan has no commitment period, this can act, if nothing else, as a stop-gap until we can get the antenna set up.

So, we shall see how this works....

XenServer Network Bonding

I've just had a bit of a learning experience with respect to Citrix XenServer and what it refers to as network bonding, and thought I would share here.

All of what I am about to say comes with the disclaimer that I am not a network engineer, but rather a server engineer, so my networking terms may not be 100% accurate.

I've been working to deploy a new XenServer pool here at $WORK recently, and we've been working under the assumption that we would connect it to our new datacenter networking standard, which we refer to locally as a "virtual chassis" -- two in-rack switches "stacked" together connected independently via 10GE to two stacked aggregator switches and network traffic tagged with standard 802.1q vlan tags.  In order for this to work as a fault tolerant configuration, each server must have a connection to each of the in-rack switches.  On our normal, non-virtualized linux server deployments we have been using standard link aggregation (otherwise known as bonding or NIC teaming) with LACP (Link Aggregation Control Protocol).

According to the documentation, XenServer, which is really just Linux under the hood, supports network bonding, so this should be the same, right?

Wrong.

Rather than using LACP, XenServer actually uses something called "Source Level Balancing," which is based on the standard Linux "Adaptive Load Balancing" method of bonding (See Citrix's KB article CTX124421).  The really cool part of this bonding mode is that it requires no knowledge on the switch side of the connection to make it work.  Instead, the hypervisor migrates VM network traffic from port to port by sending a "gratuitous ARP" through the new port with the MAC address of the VM to be moved.  In an active-active configuration, XenServer monitors the traffic for each of the running VMs, and will rebalance as needed every few seconds (according to the manual, 10 seconds).

So, don't do what I did and configure the switch ports that XenServer uses as LAG groups.  Otherwise you'll end up with a bonding mismatch and waste time trying to figure out why you are having weird networking issues.

Fixing a Galaxy Tab 10.1 "Infinite Reboot" State

I've had this documented in my personal notes for awhile, but it dawned on me today that perhaps others out there may benefit from my experience, so I am going to post this publicly.

Last year Google provided attendees of Google I/O a very nice tablet -- the Samsung Galaxy Tab 10.1, not to be confused with the retail version of the 10.1, this edition has the "army of androids" on the back, and has a slightly different version of Android on it.  Unfortunately, it also seems to have a habit of occasionally getting stuck into an "infinite reboot" state.  The tablet will freeze, restart, run through the initial startup animation, pause, and then restart again.  Repeat ad nauseam until the battery dies.  I've had my tablet get into this state twice since I got it.  Rumor has it, if you report the problem to Samsung, it is a ship-back-to-the-manufacturer type of problem, but if you have a few technical skills, this may not be necessary.

Mobile blogging

Huh, look at that.  Mobile blogging.

Trying this again

It seems like every few months I get the idea that it might be handy to have a blog.  This is just the latest iteration.