osl

Quite a few of the projects hosted at the OSU Open Source Lab are using Xen virtual machines. If you are associated with one of those projects you may be interested to know what exactly our current setup is and what my future plans for it are. If you are not currently hosted by us maybe you will be some day. :-)

Since last fall the I have been running a Xen cluster at the OSL which is slowly replacing our original independent Xen hosts. We currently host a total of 41 Xen virtual machines which include projects like Busybox, Inkscape, an OFTC IRC node, the Freenode website, the OLPC user support forums, and many others. Currently 17 of those are on the new cluster split between 3 of the 6 available host nodes. The other 24 virtual machines are still on two of our older independent Xen hosts.

The older Xen hosts are just boxes loaded with lots of disk and ram, with the virtual machines running off of the local disk space. The problem with this setup is that Xen and Linux kernel upgrades are incredibly difficult. Since the virtual machines cannot easily move to another host upgrading Xen requires taking an outage for all 8 to 12 virtual machines running on that host. To complicate matters Xen can be a bit troublesome to install/upgrade sometimes so it is not uncommon for such an upgrade to take much longer than expected. To improve this situation I built out our Xen cluster.

The cluster currently consists of 6 Xen hosts which are part of a 14 blade IBM Bladecetner that was donated to us by Intel. The 6 hosts each have 4GB of RAM and dual Pentium 4 processors and can typically run between 6 and 8 virtual machines depending on RAM and CPU needs. The remaining 8 blades will eventually be built out as more hosts but currently are waiting on RAM. (Anyone have a pile of 1GB PC2100 sticks laying around?) All of the disk space is hosted via iSCSI on a separate disk node. The current disk node is a Dell 2650 with 260GB of disk for virtual machines and is serving up that space with ietd since we don't have a hardware based iSCSI target card.

The good thing about this new setup is I can migrate virtual machines between host nodes on demand while they are running so I can easily upgrade the host nodes as needed. Maybe some day I will get better monitoring set up so I can move virtual machines around to balance CPU load but that's not planned for the near future. The bad thing is I still have a single point of failure with the single disk node. Also the disk node doesn't have very much disk space so we have nearly filled it up which is why the cluster is only running 17 virtual machines. So the setup is not perfect but it's a pretty good start using hardware that was either donated or we already had.

Down the road I want to replace the current disk node with two boxes replicating the data using DRBD and set up graceful fail-over between the two using heartbeat. The current plan is to upgrade the disk space on our mirror servers and use some of the old disk arrays for the Xen cluster. This will give us about 3TB total for 1.5TB of redundant disk space between the two disk nodes. That will give us enough space to move all of our existing virtual machines over to the cluster with room for 30-40 more for a total of 54-64. That won't quite fill up the Xen host nodes which can probably host 80-90 virtual machines while keeping one host node as a hot spare. It will be enough room for about a year and a half worth of growth and should enable us to provide great up time for the hosted projects. :-) Unfortunately with this plan the Xen upgrade is waiting on the mirror upgrade which is waiting on money to buy the new disks and I have no idea when that is going to happen. Hopefully something will pull though soon, the mirrors have been needing this upgrade for nearly a year now.

And how does Gentoo fit into all of this? All of the Xen and disk hosts run Gentoo and are managed by our central cfengine system. I have been maintaining the Xen packages for Gentoo to keep them in working order for use at the OSL and the whole setup seems to work pretty well now. Hopefully later today I'll have a chance to start rolling packages for Xen 3.1.3 and 3.2.0.

So, here is a crazy idea that I actually got around to implementing: The OSL Beer Fund. As a sysadmin for the Open Source Lab I work with a lot of people via email and IRC from all over the world. As a student I don't exactly have much of a budget to travel to far off conferences to meet up with many of those people. So the idea is this: if you would like to help save our sanity or simply say thanks to one of us but can't meet up for beers in person why not contribute a couple bucks to our beer fund? In return we can even give your favorite server a little extra luvin'. Just contribute via paypal to beerfund_AT_osuosl.org.

Disclaimer:
The OSL Beer Fund is not associated with the OSU Open Source Lab in any official way. It just happens to have the email account 'beerfund' under the osuosl.org domain, by chance a random letter generator gave us the acronym 'OSL' to use as the fund's name, and somehow all beer purchased with the fund manages to go to students who work for the Open Source Lab (and who are of age of course). How that all worked out I don't know, we sure didn't plan it. ;-)

Basically, I need a smaller laptop. My current one is breaking, heavy to carry around, and most importantly the screen blocks to much of the fire that I am sitting in front of at the moment. Go go back yard fire pits! :-)

Over the past month I have been working full time at the OSL and have been getting quite a bit done which is a very welcome improvement over the school year. I think I accomplished more in the first week of the summer than I did over the entire previous term. I can hardly remember what all I did, but the biggest things have been rewriting our internal inventory app and Xen stuff.

Our original app was written in Ruby on Rails, was slow as heck, and adding stuff required figuring out a new and strange looking language. (And after our experience with trying to run a busy RoR site none of us sys-admins wanted anything to do with it anything in RoR.) Over a couple weeks I managed to plop a new CakePHP based interface on top of the old database, add a handful of new features like automatic database schema upgrades, a simple visual view of each rack, and the whole thing runs a lot faster than the old one did. The app mostly does what we need it to so I've stopped development, any fancier features can wait until RAIV is done.

On the Xen front, I spent part of last week getting the 3.1.0 ebuilds ready to go along 2.6.18 and 2.6.20 kernels! Finally no more 2.6.16! :-) The only significant issue that I know of at this point is that the 2.6.20 kernel will not run as a x86_32p guest on a x86_64 Xen, but since support for that is new to 3.1.0 and the 2.6.18 kernel is working I'm not going to worry about it to much. So unless something significant crops up I'm going to push it all into the portage tree early next week.