On 15 Jul 2005, at 1:53 pm, Tim Cutts wrote: > > On 14 Jul 2005, at 11:37 pm, John H. Lee wrote: > > >> >> Since Juan's system is OS X, can anyone offer a suggestion for OS >> X? Cluster filesystems like Lustre and PVFS sound promising for >> Linux, but what options do we have on Macs? Any experience with >> Xsan? With a fibre channel limit of 64 endpoints, can Xsan even >> be considered for systems as large as Tim's? >> >> We have a 16-node Xserve cluster with one Xserve RAID. The RAID >> is connected via FC to one server, which exports the volumes via >> AFP and NFS over gigabit ethernet. As expected, the AFP/NFS >> server is a bottleneck. >> > > I believe SGI's cluster filesystem, CXFS, offers client access for > Mac OS X. It's probably in the works for Lustre too (OS X's > filesystem plugin architecture should make an open source effort > possible - I just don't know whether the Lustre guys have done it) > I should have said - to answer the first parts of your question, I don't know what the scaling limits of most of these filesystems are. Currently, we keep them quite small; 14 or 28 nodes per cluster filesystem, using GPFS. That still means we only have to distribute a dozen copies of the data for our first IBM farm, and those 12 copies are then shared by 168 dual CPU machines. That's a lot better than having to use 168 separate copies, or attempting NFS. I have a feeling Xsan's limits are quite small - 16 nodes rings a bell, but someone from BioTeam will probably know the answer to that question. Great things are promised for Lustre, and I guess we'll find out over the next few months just how far we can push it. We intend to use it for cluster-global scratch filesystems first. (a) these get battered most heavily (by concurrent writes, too, the worst of all things), so will be a good test, and (b) it's scratch, and the users know it can go at any minute, so we'll be OK in case of filesystem disaster in this new and unknown territory. Tim -- Dr Tim Cutts Informatics Systems Group, Wellcome Trust Sanger Institute GPG: 1024D/E3134233 FE3D 6C73 BBD6 726A A3F5 860B 3CDD 3F56 E313 4233