[Pipet Devel] toolkit and data access/storage

[Carlos Maltzahn]

Correct me if I'm wrong but I think what we really are trying to design
here is somekind of batch processing management system. Our department
runs a commercial product called Load Sharing Facility (LSF) sold by
Platform Computing (www.platform.com). See
http://www.cs.colorado.edu/csops/FAQ/lsf.html for our installation and
http://www.cs.colorado.edu/csops/FAQ/lsf-webpages/quick-admin.html for
some documentation on it.

Is there an open source equivalent for this? 

If we want to define and monitor computations with GCL we need to have
some way to manage batch processing on different machines, query their
state, and have access to intermediate results or check points. That means
each execution needs to be submitted to some kind of management system
that then schedules and runs the tools in some kind of shell that it can
remotely query and control. Once we have established such a management
system it should be fairly easy to write a GCL user interfaces for it. I
can see Paos to sit on top of such a management system and the GCL editors
and monitors to be Paos clients.

LSF is designed primarily for workload management. Our focus would be more
on composing tools, and scheduling, and controlling them.

Carlos

On Thu, 21 Jan 1999, J.W. Bizzaro wrote:

    on the server side, we are catering to
    fork-request-once-response-once-terminate programs made by
    who-knows and whenever with whatever language.  In other words, we
    still need a CGI-like system.  But this is only one type of Loci
    client.  Other clients can make better use of Paos.
    
    > I personally don't like CGI 
    > because of it's unflexible fork-request-once-response-once-terminate 
    > assumption. I suspect these tools are running for a longer time period and
    > we would like to be able to find out about their state.

    Yes!  This is something I realized would not work with standard
    CGI.  These analysis algorithms (server side only) will be longer
    lived than standard CGI scripts.  Some may take hours or days to
    complete.  I would like to return the state or maybe the time
    elapsed to the client so that the user knows something is
    happening and the whole thing didn't just die.
    
    > On the other 
    > hand it's probably not a good idea to run them natively in Paos 
    > because of their size, no? What are people's thoughts about this?
    
    Because biological analysis algorithms tend to be command-line,
    run-once-and-terminate, I think we will just have to treat them
    like rather big CGI programs.  That's not to say that other tools,
    libraries, etc., written for Loci, will not communicate directly
    using Paos and XML...They can and will.
    
    
    Jeff
    -- 
    J.W. Bizzaro                  Phone: 617-552-3905
    Boston College                mailto:bizzaro at bc.edu
    Department of Chemistry       http://www.uml.edu/Dept/Chem/Bizzaro/
    --