[Bioclusters] Re: Bioclusters Digest, Vol 9, Issue 8

M. Michael Barmada barmada at pitt.edu
Fri Jul 15 11:57:20 EDT 2005

On 7/15/05 9:05 AM, "John H. Lee" <johnhlee at berkeley.edu> wrote:

> Since Juan's system is OS X, can anyone offer a suggestion for OS X?
> Cluster filesystems like Lustre and PVFS sound promising for Linux,
> but what options do we have on Macs?  Any experience with Xsan?  With
> a fibre channel limit of 64 endpoints, can Xsan even be considered
> for systems as large as Tim's?
> We have a 16-node Xserve cluster with one Xserve RAID.  The RAID is
> connected via FC to one server, which exports the volumes via AFP and
> NFS over gigabit ethernet.  As expected, the AFP/NFS server is a
> bottleneck.

As I mentioned, we're running Xsan in our OS X cluster, and also for our
department-wide fileserver (also OS X). This is absolutely an improvement,
as we can distribute the file sharing load over multiple NFS servers, all of
whom can export the same volume. For our 120 node Xserve cluster, this is an
immense help, as now each NFS server responds to only 30 clients, rather
than trying to serve 120. Initial testing we did (see
http://www.bioitworld.com/archive/121504/box.html) indicated that the
reduction in load is nearly linear (i.e. 4 NFS servers gives us a 4x
increase in performance over a single NFS server for all 120 cluster nodes).
This is really great news, and we could do even better by adding additional
NFS servers, but setting up the Xsan array properly is a bit complicated,
and not as inexpensive (to do things properly) as might be initially thought
(we were pretty early adopters of this technology, and so found out the hard
way what not to do). In particular, its important to have a private network
for the metadata communications, and its best to isolate metadata to its own
dedicated storage pool (as Xsan really hits the metadata storage containers
hard and generates a lot of communication traffic). This can bump up costs

We played with PVFS and other similar shared filesystems (MFS, GFS) a while
back on a smaller (48 processor) linux cluster, but never really found
anything that was as robust (as NFS) for processes that did a lot of I/O and
clusters with lots of nodes. I imagine the situation is somewhat better now,
but I don't know which (if any) of these have been ported to OS X?

M. Michael Barmada, Ph.D.
Associate Professor of Human Genetics
Graduate School of Public Health, University of Pittsburgh

            There are three kinds of people in this world:
             those that can count, and those that can't...

            The requirements said: Windows 2000 or better.
                        So I got a Macintosh.

    To know the mighty works of God; to comprehend His wisdom and
  majesty and power; to appreciate, in degree, the wonderful working
 of His laws, surely all this must be a pleasing and acceptable mode
   of worship to the Most High, to whom ignorance can not be more
                      grateful than knowledge.

More information about the Bioclusters mailing list