On 20 Jan 2005, at 11:47 pm, Speakman, John H./Epidemiology-Biostatistics wrote: > Ten HP Proliant nodes, one DL380 and nine DL140. Each node has two > 3.2Ghz Xeon processors. They do not have a dedicated switch; the > infrastructure folks say they want to implement this using a VLAN. We > have some performance concerns here but have agreed to give it a try. You might well end up with performance problems unless the nodes have plenty of local storage so that the jobs don't have to hammer NFS. VLANs are great, but just wait until the infrastructure people start whining that the backups are failing because the switch is full of your cluster traffic. :-) > > User characteristics: > > The users are biostatisticians who typically program in R; they often > use plug-in R modules like bioconductor. They always want the newest > version of R right away. Also they may also write programs in C or > Fortran. Data files are usually small. Nothing fancy like BLAST, > etc. I can see why that gives you a leaning towards Debian. > User concerns: > > Users require a Linux clustering environment which enables them to > interact with the cluster as though it were a single system (via ssh > or X) but which will distribute compute-intensive jobs across nodes. > As the code is by and large not multithreaded, it is expected that > each job will be farmed out to an idle compute node and probably stay > there until it is done. That’s fine. In other words, to use all > twenty CPUs we will need twenty concurrent jobs. Getting R to work seamlessly on a cluster is not trivial, unless the users are using R in batch mode. Getting R to work interactively and still have the graphical parts of it work correctly is less easy. X authentication starts to be a problem. It's unfortunate that R doesn't have some sort of client-server architecture which would allow the hard work to occur on a cluster node, and the interactive stuff be handled locally by a client. > Administration concerns: > > The cluster must require the absolute minimum of configuration and > maintenance, because I’ve got to do it and I’m hardly ever around > these days. cfengine is good for this, as you've already suggested. We use cfengine for managing configuration information on our Linux desktops, although not on our cluster nodes. > Other concerns: > > Users and administrators alike have a preference for Debian Linux over > other distributions. How nice to have such clued-up users. I envy you. > Potential solutions: > > We like the look of NPACI Rocks but its non-Debian-ness makes it a > last resort only. What we would really like to try is a Debian > version of NPACI Rocks; in its absence we will probably have to use > two separate packages to fulfil the requirements of #1 and #2 above. > > Sensible options for #1 seem to be: > (1) SystemImager (www.systemimager.org) > (2) FAI (http://www.informatik.uni-koeln.de/fai/), maybe also > involving the use of cfengine2 (http://www.iu.hio.no/cfengine/) > > SystemImager is the better-established product and looks to be simpler > to set up than FAI and/or cfengine2, in both of which the learning > curve looks steep. However, FAI seems more elegant and more like the > idea of “NPACI Rocks Debian” that we’re looking for, implying that > once set up FAI/cfengine2 will require less ongoing maintenance. We use cfengine to configure both Red Hat and Debian machines here. We use FAI for Debian desktop installs (we're in the process of converting all desktop Linux boxes from Red Hat 9 to Debian). Yes, it does have some difficulties to start with, but it's *really* fast. A node about the same speed as yours goes from bare tin to installed and running in about 2 minutes. We have a few Debian developers on site (I'm one of them), and one approach that really helps here and is easy to set up is that we now have a local APT repository for packages created locally, including in-house packaging of proprietary or in-house software to make it easier to splat out across hundreds of machines. FAI can be trivially pointed at such a repository as well as our main Debian mirror. > > Sensible options for #2 seem to be: > > (1) OpenMosix > (2) OpenPBS > (3) Sun GridEngine N1 > > Note: all of the above have commercial versions; we’d be reluctant to > consider them unless it means big savings in administration time and > effort. We get the impression OpenMosix (and, to a lesser extent, > OpenPBS) have question marks over how much time and resources the > people maintaining these products have, suggesting bugs, instability > and not keeping up with kernel/library updates, etc. Sun GridEngine > seems more robust but does not seem to have a big Debian user base. GridEngine is pretty good. I think the lack of Debian support is probably a licence issue - I'm not sure SGE is DFSG-compliant. As others have said, MOSIX or OpenSSI will probably be easier for your users to cope with, especially if they want to run R interactively. At the end of the day, cluster users can't remain ignorant of the implementation of the systems they are using; if they remain ignorant, they will do things which (a) run very slowly and/or (b) bring the cluster or the network down. Accidental NFS abuse being the most common way they do this, in my experience. Tim -- Dr Tim Cutts Informatics Systems Group, Wellcome Trust Sanger Institute GPG: 1024D/E3134233 FE3D 6C73 BBD6 726A A3F5 860B 3CDD 3F56 E313 4233