[Pipet Devel] batch processing

[Jarl van Katwijk]

> > I think the sequence or hiarchy of the nets are something of high importance to a
> > batch schedular: job that should not start untill other(s) are finished. Or job
>
> Wouldn't all those dependencies be solved by "pulling" on the data-flow? or am I
> missing something?
>

consider this structure:

>--O1
          \
>---O2--O5---O6-->
           /       /
>--O3     /
               /
        O4/

Where the O's are the nodes. Once could execute on dataflow basis, so O1,2 & 3 will run,
and before O5 O4 has to run, so O5 will stall untill O4 is ready.
As good batching will run O4 before O5 is stalled. But I must admit the dataflow is 90%
of the batch logic.

>
> > termintion once certain situations occure.  I've worked a lot with JCL, job control
> > language, with is the scripting language of IBM mainfraimes. A very typical aspect
> > of jcl is that it's laking iterations and jumps.
> > So no for\next until\while stuff and no goto's.  Does Loci depends upon those or
>
> Loops are supported by Overflow through the "Iterator" (which derives from the
> subnet). I'm also using some kind of "checkpointing" in Overflow for very long
> jobs (a couple days of processing), such as neural network training. Now, about
> goto's, I'm strongly against them in a data-flow type of processing.
>
> > suggest we'll ALWAYS use timeouts on jobs and use triggering or stalling as the
> > only scheduling primals.
> > I'm only talking about SUBNET batching, inside those at PL level nothing will
> > change.
>
> I think most (though not all) of this is likely to be simpler if implemented in
> the PL.

You did take cpu load, ram\HD space usage etc into account already?

>
> P.S. Brad and I have ran into some problems with the antialiased gnome canvas as
> used in Overflow (But everything now *compiles* on FreeBSD). Has anyone compiled
> Overflow on something else than Mandrake 7.0 and FreeBSD?

I had it compiling on my slackware 7.0 with gcc 2.95.x manually installed. But I
reinstalled slack :(