Pull-based job scheduling (was: [Bioclusters] Best ways to ta ckle migration to dedicated cluster/Farm)

[Chris Dwan (CCGB)]

> I have also successfully implemented a RDBMS-based pull system that worked
> very well.

Having written one myself, I have respect for all the home grown workload
management tools out there.  I've seen them vary from combinations of cron
and "at", with shared files for job allocation, all the way up to the
RDBMS / thin client solutions.  Mine was an rsh based push scheduler
implemented mostly in Tcl using flatfiles for state.  It kept 5 quad
processor P-II's busy pretty much full time running BLASTs for about three
years.

Here's a question for those who have created homemade workload managers:
Would you do it again, today?  Why or why not?  Personally, I would try
every other avenue before writing another scheduler.   Home grown systems
tend to make the developer a critical resource and a single point of
failure.  It's sort of like implementing your very own database management
system.  Maybe fun for the developer, but bad and wasteful for the
organization.  You can get all the power you want out of commercial /
and open source solutions for DRM and DBMS problems.

That said, there still are problems where home-grown is still the best way
to go.  In my opinion, one of them is stitching together compute resources
across organizational and administrative boundaries.  For which other
common tasks do people think it's still cost / time effective to build
homebrew solutions?  What homegrown software do you rely on, but dream of
replacing with someone else's supported code?

-Chris Dwan