[Bioclusters] Request for discussions-How to build a biocluster Part 2 (hardware)

[Donald Becker]

On Thu, 2 May 2002, Sylvain Foisy wrote:

> So,let's start. A reminder: this is coming from a total newbie at this
> BioCluster stuff. it is also to serve as the seed of a tutorial site for
> building the beast.

The hardware side is extensively covered on the Beowulf list, often with
excellent posts from people such as Robert Brown.  Reading just RGB's
posts for the past year will make you a cluster expert.

Many of the Beowulf list people are interested in floating point
intensive simulations, so there is some difference from current life
sciences applications.

> -> CPU: I have seen some debate on this list about PIII vs Athlon as
> CPUs, especially about power requirements as well as the heat outputs.

The usual recommendation is to stay a half generation behind for best
price-performance.  However some memory bandwidth intensive applications
perform much better on the latest P4 systems.  Applications that are
bound by memory latency perform about the same on the P3 and P4.

Athlons and P4 both have about the same peak power use, while the Intel
chips use much less average power.  This might be a consideration for
average A/C load.

> -> RAM: We are inclined to load the max possible for our selected
> motherboards (Tyan) at 4Gig. DDR-RAM of course. Not much debate here.

If your application fits in less than 2GB, you will get better
performance with only 2-3GB of memory.  Linux kernels 2.2 and earlier are
limited to that amount, and the 2.4 kernels must do extra work to handle
more memory.

Note that 2.4 kernels before 2.4.17 will perform very badly with
multiple 2GB jobs run in series.  Be certain to use 2.4.17 or 2.4.18 for
large memory jobs.

> -> HD space: If I read the list archive right, space is needed so that
> parts of GenBank (or any other DB) can reside in each node for BLAST to
> do its thing. The choice is either to use built-in ATA-100 interface to
> 40GB 7.2K drives or optional SCSI interface for a 36GB 10K drive(our
> selected options right now).

If you have more memory than your data set, the disk I/O rate isn't
critical.  Our users typically configure  1.5GB or 2GB of memory on
compute nodes, and use a 600MB primary data set.  After the first
search, the data set is almost entirely in the page cache.

This isn't a reason to use the cheapest 5400RPM disks.  There is a
negligible price premium for performance-oriented 7200 ATA-133 disks.

Note: If your application really does justify 10K RPM drives, count on
at least triple the heat load.  That means more fans and reduced density.

> ->Interface card: Yeah, Myrinet would be great. Back to Earth: we are
> thinking either 100Base-T or Gigabit Ethernet (most probably the first
> one).

The communication needs are very application specific.
Our hardware integrators are now typically shipping Gb NICs so that the
cluster can be upgraded with just a switch swap.

> Can we work with the built-in NIC interface of the motherboard or
> should we get PCI NIC cards?

On-board NICs are PCI connected, so there is not performance difference.
You might expect that on-board NICs have network boot support, but
that's frequently not true.

> ->Networking gear: all this will be linked together to the head via a
> rackmounted switch. I heard great things about the Cisco stuff. How
> about 3Com? Other choices out there?

It depends on the cluster size.  Large Gb switches are rare and
expensive, pushing people to use Myrinet on the large clusters.  For
fast Ethernet other vendors to look at are Extreme, Foundry, and HP for
>32 ports.

> ->Enclosures: all rackmounted with 2U enclosures

From the cooling viewpoint, 2U is best for dual high-end P4 or Athlon,
other configuration have acceptable cooling in 1U.
2U systems are almost always less expensive than 1U.

> We intend to start with 8 nodes and planning to
> build it up to 32 or 48 enclosures (total of 96 node processors).

With only 8 nodes, inexpensive Gb switches are readily available.
You might plan on just replacing the switch when you add nodes.

> ->Physical installations: we are no engineers so what would we need, in
> AC and electrical power  inputs? We are planning to retrofit a former
> office into a server room. Anybody with similar experience?

Power is cheap and easy to retrofit.  A/C is frequently the limit.  You
might end up designing your cluster around the available cooling.  It's
possible to trade $$ for lower BTUs.

-- 
Donald Becker				becker@scyld.com
Scyld Computing Corporation		http://www.scyld.com
410 Severn Ave. Suite 210		Second Generation Beowulf Clusters
Annapolis MD 21403			410-990-9993