[Pipet Devel] Re: Overflow/Loci/GMS collaboration

[J.W. Bizzaro]

Jean-Marc Valin wrote:
> 
> > > 1) Wrap all Overflow as a Loki/GMS node
> > > 2) Use Overflow's core and implement the missing functionnalities.
>
> OK, let's talk about 2) (I consider 1 to be trivial). All our nodes are classes
> and the code that's ran is in the getOutput method. Maybe it's possible to call
> the GMS functions from there. Again, the main reason I'd like 2) is speed: not
> having to call an executable for a sum, logical and, even an FFT.

The big problem I see with approach 2) is that Overflow has no knowledge of
where it is in a physical network, where other instances of Overflow might be,
what nodes are available at what location, how to request a particular remote
node, and how to send data between two remote instances of Overflow without it
first coming back to the local instance.  These are all problems Loci was
designed to deal with.  But if you're talking about Overflow being the real
core, as core it would have to deal with these problems.

So, I don't think plugging GMS into Overflow's port will make Overflow/GMS a
distributed system.  As an example, let's say we have computers A and B.  A is
the local computer run by the user, and B is a remote computer.  Both A and B
have copies of Overflow/GMS, except computer B has a node that computer A does
not have.  Now tell me, how will Overflow/GMS on A know (1) where computer B
is, (2) what nodes are on B and (3) how to request access to the node?

Alright, let's make the problem a little more complicated.  We have 3
computers: A, B and C.  A is again local while B and C are separate remote
computers.  Now, the user on computer A wants to pipe data from a node on A,
to a node on B, and then directly to a node on C.  How does (4) Overflow/GMS
on B know to send the data to C and not back to A?

The only way I see to address these problems, and what Loci was proposing to
do, is create a directory service and a meta-data transfer system.

In Loci, each instance of Loci's core is identical to the next, regardless of
location.  And each has the ability to contact another with a request.  If,
for example, a node exists on computer B that the user wants (access to), Loci
on A would ask Loci on B for it.  And the system is very much seamless and
symmetric: If a node exists on computer A that the user wants, Loci on A (same
computer) ASKS ITSELF FOR THE NODE.

And if the user wants to connect a node on B to a node on C, Loci on A sends B
the request with meta-data saying to contact C when done.  Next, B contacts C,
and then C contacts A that the connection was made.

> > So, Loci would allow the user to execute a node and THEN start adding
> > connections...or even change connections at a later time.  Loci does this by
> > making a 'dynamic batch file', which really has to be a database (since we are
> > using XML, we were going to use an XML database).
> 
> I'm not sure I understand what you mean here?

I mean that all of the information for each node (including meta-data) and all
of the connections made between nodes, are repesented as XML.  BUT, to make
parsing and editing this XML structure easier, we put the information into a
database.  The information is actually transformed from XML into a form better
suited for the database, and then when retrieved, the information is put back
into XML, which is better suited for transfer and storage.

> You can have a set of nodes that generate a command line in Overflow, then feed
> that command-line (string) to the "Exec" node.

Great.  That's pretty much what we were trying to do, but we have some rather
interesting rules for construction that I'll have to pass on to you.

> Is the "program" in Loci stored as a script?

What are you calling 'program'?  If you mean the network to be ran (or
workpath to be followed in Loci-speak), then yes, it's stored as a script...in
a database.

> The way it works in Overflow, is
> that the XML is simply storing a set of objects: Networks, Nodes and Links.

Exactly the same as Loci!

> It
> stores how the nodes are connected together (XML was just a convenient way to
> serialize our objects).

Right.  Except, for doing dynamic editing and parsing, we need to take the
data out of XML and put it into a database.

> Another thing you need to know about Overflow. It works
> using a "pull" on the nodes. That is, you first ask the last node for its
> output. Then this node asks for the value of each of its input nodes. These
> nodes do the same until everything is calculated. If, for some reason a value is
> unused, it is not calculated.

Excellent!  This is exactly how we planned on 'running' a network.

> Dominic has a couple things he wants in Overflow (or anything it becomes):
> Asynchronus events, multi-thread stuff (partially implemented), ... (Dominic,
> anything I forgot?)

Great.

> Sounds cool! I'm not really a GUI guy. The vflow GUI is about the best I can do
> (I actually learnind GTK/GNOME while writing it).

I think I am a GUI guy afterall.  I tend to need to see the results of my
work.

> I use Overflow for speech processing/recognition and Dominic uses it for image
> processing and robot behaviours. I'll probably also write a real-time music
> effect processor... when I have time.

Oh cool, a couple of AI/robotics guys.  I think our needs are indeed very
similar.

Cheers.
Jeff
-- 
                      +----------------------------------+
                      |           J.W. Bizzaro           |
                      |                                  |
                      | http://bioinformatics.org/~jeff/ |
                      |                                  |
                      |        BIOINFORMATICS.ORG        |
                      |           The Open Lab           |
                      |                                  |
                      |    http://bioinformatics.org/    |
                      +----------------------------------+