I've had a blast the past few days doing rack-and-stack work that I normally don't get to do much anymore. Rough notes and a link to the images follow... The pictures: ------------- http://bioteam.net/gallery/wetlabcluster The challenge: -------------- In 12 days or less, design a cluster, source the parts and put it together in good working order. The cluster must meet the following requirements: - Capable of operating in a wet lab setting - Managed and operated by biologists - Linux OS required (software dependencies ...) - Require no more than 2x 20-amp power circuits - ~ 4 terabyte raw storage requirement; HA or super-performance not a requirement - Quieter than the instruments surrounding it - Small enough to (roughly) fit under a lab bench - Have sufficient CPU power to meet analytical needs - Capable of automatically processing data coming off one or more high end instruments The components: --------------- I can't share details about the requirements gathering phase of the project. We studied the instrument, the science and the stuff that needed to be done with the data coming off the instrument and determined that approx. six dual-processor boxes with AMD Opteron CPUs would be acceptable. Under a massive time crunch and some components were ordered purely on the basis of "how fast can you ship to us..." The parts list boiled down to the following pieces: From CDW.com with rush delivery :) - Digi CM 16 serial console server - Pair of 20-amp APC rack-mount power distribution units - Dirt cheap SMC 24-port gigabit ethernet unmanaged switch - Box of serial DB9 to cat5 RJ45 adaptors for serial console - Bulk quantities of 5ft grey cat5e cables (no time for special colors or lengths) From IBM via a local reseller/integrator: - 7x IBM eSeries 326 1U rackmount dual-Opteron servers (6 nodes + master) From Apple: - Apple Xserve RAID with 14x 400gb drives - Apple PCI-X fiber channel HBA card & cables - Xserve RAID spare parts kit From Extrememac.com: - Small form factor 12U "XRack Pro2" cabinet (http://www.xrackpro.com/) The problems: ------------- The biggest overall problem was that the Apple Xserve RAID was ordered with Fedex shipping but without priority delivery. This means that the storage arrived at 5pm the night before our final cluster-assembly work day. It also arrived with damaged rackmount rails but the damage was not enough to make the hardware unusable. Even worse, the cluster cabinet arrived at 1pm *on* our final work day. This was in spite of the fact that the cabinet had been ordered via credit card directly from Extrememac 7 or 8 days prior. As a vendor, they were not really on the ball with things but this could be normal for a company that seems to mostly make iPod accessories. Hopefully just a fluke experience. The IBM hardware arrived quickly and the reseller/integrator did a good job. A minor hassle was that we had to order 15,000RPM Ultra320 scsi drives because the cheaper 10,000RPM drive were on some sort of IBM global "short supply" list. The biggest problem with IBM and the reason I'll probably never purchase eSeries servers again is that apparently IBM refuses to sell any sort generic rail mounting kits for the e-series product line (this is what the integrator told me; have not verified this yet). They ship with rail kits that *only* work in IBM branded server cabinets. Given that we were installing into a non-IBM 12U cabinet this was a big issue. Our integrator found a 3rd party rail reseller that makes compatible rails but we could not order them in time. To me this is just annoying and (if true) due to the annoyance factor I'll probably buy my dual Opterons from Sun in the future (assuming Sun will sell me a generic rail kit...) Final thoughts: --------------- The 64bit version of Suse 9.2 Professional handled the fibre channel storage amazingly cleanly. It detected, mounted and provisioned the 2 Apple RAID LUNs into a LVM group with no problem at all. I was expecting the Linux -> Apple RAID stuff to be a bit more scary. I really like the XRack Pro2 cluster cabinet or whatever it's marketing name is. Well assembled with good options for choosing between quiet vs cooling. There is plenty of space for wiring and cable runs even if all 12U are packed with equipment. We have everything powered up today and working hard and are monitoring the temperature conditions internally. The Xserve RAID is one the quietest storage arrays I've ever seen - I thought it would be louder than the IBM rack-mounts but this is not the case. The biggest liability in this cluster is the lack of an internal UPS capable of cleanly shutting down the Xserve RAID chassis. There was simply no more room in the cabinet. We'll do external UPS for now and if we can squeeze out 1 compute node there is the possibility of installing one of the 1U UPS systems made by APC. -Chris -- Chris Dagdigian, <dag at sonsorol.org> BioTeam - Independent life science IT & informatics consulting Office: 617-665-6088, Mobile: 617-877-5498, Fax: 425-699-0193 PGP KeyID: 83D4310E iChat/AIM: bioteamdag Web: http://bioteam.net