[Pipet Devel] batch processing

[J.W. Bizzaro]

Jarl van Katwijk wrote:
> 
> Before I'll start being too technically about batch processing, I want to know
> if people have a wish list for this. I haven't looked into detail at these
> projects, but I've some professional experience with batch processing (SA7).
> Personally I do not have any speciafic wishes, but I see why the bioinformatic
> people need to have this, thnx to Jeff's explainations about the linking of
> simulation software inside the Piper environment.

For those not familiar with bioinformatics and our plans for Loci, we're
talking about the difference between sequential (non-streaming) and
non-sequential (streaming) data transfer.  A bioinformatics (or scientific)
computation is typically performed "all at once", returning results when
finished.  Many are often long-running (depending on the computer), taking
minutes, hours, or days.

While designing Loci, my first thought was that these "jobs" could be handled
as they are on a supercomputer: with batch processing.  This allows some
information to be returned about the progression of the job, cpu time used,
etc.  It also allows jobs to be scheduled to run at a certain time, with a set
priority, etc.

You may think that batch processing has disappeared along with expensive
computer time.  But I think it may be particularly important for Piper,
because (1) the system needs to report back to the user when the job has been
"gone" for a long period of time (or else the user may think the data has been
lost) and (2) because we Piper will use "other people's computers" (we want
those people to decide when jobs will be run and to decide the resources to be
used).

Thoughts?

Cheers.
Jeff
-- 
                      +----------------------------------+
                      |           J.W. Bizzaro           |
                      |                                  |
                      | http://bioinformatics.org/~jeff/ |
                      |                                  |
                      |        BIOINFORMATICS.ORG        |
                      |           The Open Lab           |
                      |                                  |
                      |    http://bioinformatics.org/    |
                      +----------------------------------+