[Pipet Devel] Re: Overflow/Loci/GMS collaboration
J.W. Bizzaro
bizzaro at geoserve.net
Tue Mar 14 01:16:27 EST 2000
Jean-Marc Valin wrote:
>
> The reason I was suggesting the Overflow core is that we already handle IO with
> files and pipes, so Overflow nodes can also be executables (it's tested and it
> works). From what I know now, I see two options:
> 1) Wrap all Overflow as a Loki/GMS node
> 2) Use Overflow's core and implement the missing functionnalities.
If we chose 2), what would we use GMS's core for? I'd like to hear Jarl's
thoughts on Overflow's core. GMS may provide the missing functionality for
Overflow, and merging C and C++ shouldn't be all THAT difficult. I was
thinking about 1), but sure, we should consider 2) as well.
> I'd personnaly prefer 2) but the main question is whether the design we have can
> handle the new features from Loci/GMS. I think it is general enough but again,
> you might prove me wrong. Can you try find out things in Loci you don't think
> Overflow can do, and I'll try to see whether that's the case.
Okay, here's one: Overflow runs from a batch file created by the GUI, right?
Loci's design (and I think GMS's too) lets the core work while the user
continues to operate the GUI. This is necessary for Internet-distributed
nodes running on large datasets, because there can be a very long delay before
anything is returned.
So, Loci would allow the user to execute a node and THEN start adding
connections...or even change connections at a later time. Loci does this by
making a 'dynamic batch file', which really has to be a database (since we are
using XML, we were going to use an XML database).
How does Overflow start an executable as a node? You have to feed it a
command-line statement, right? I mentioned already that I have a neat way to
generate a command-line statement by connecting nodes :-) GMS/Jarl seems to
have developed a good system for wrapping executables. Have you seen it?
That's one way I think GMS complements Overflow, another is GMS's use of
CORBA.
I like how Overflow uses XML. Looking at some of your XML files, it seems we
were thinking along the same line. For example, here is a Loci XML script
that tells the core what connections to make ('front' is the GUI and 'middle'
is the core):
----------------------------------------------------
<front>
<add_workspace>
<workspace id='' >
</workspace>
</add_workspace>
</front>
<middle>
<success>
<workspace id='workspace1' >
</workspace>
</success>
</middle><!--mouse click at 101.0 83.0 -->
<!--clicked on workspace -->
<front>
<add>
<workspace id='workspace1' >
</workspace>
<locus type='document' >
</locus>
</add>
</front>
<middle>
<success>
<locus id='workspace1/document1.xml' num_outputs='1'
gui_name='file_select' num_inputs='1' >
</locus>
</success>
</middle><!--clicked on document -->
<!--was workspace now document -->
<!--mouse click at 184.0 73.0 -->
<!--clicked on workspace -->
<!--was document now workspace -->
<front>
<add>
<workspace id='workspace1' >
</workspace>
<locus type='viewer' >
</locus>
</add>
</front>
<middle>
<success>
<locus id='workspace1/viewer1.xml' num_outputs='0' gui_name='text_box'
num_inputs='1' >
</locus>
</success>
</middle><!--clicked on viewer -->
<!--was workspace now viewer -->
<!--mouse click at 182.0 76.0 -->
<!--clicked on viewer -->
<!--locus dropped at 259.0 116.0 -->
<!--mouse click at 100.0 184.0 -->
<!--clicked on document -->
<!--was viewer now document -->
<!--dot group xy 259.0 212.0 -->
<!--dot center world xy 259.0 212.0 -->
<!--dropped locus output line # 0 -->
<!--dropped line on mate viewer -->
<!--dropped on mate input line # 0 -->
<front>
<connect>
<input id='workspace1/viewer1.xml' >
</input>
<output id='workspace1/document1.xml' >
</output>
</connect>
</front>
<middle>
<success>
</success>
</middle><!--connecting locus document line # 0 to mate viewer line # 0 -->
<!--locus position 101.0 83.0 -->
<!--mate position 259.0 116.0 -->
<!--half_pos 180.0 99.5 -->
<!--dots xy recorded 101.0 183.0 -->
<!--locus linemovexy -79.0 -112.5 -->
<!--dots xy recorded 184.0 173.0 -->
<!--mate linemovexy -79.0 -116.5 -->
----------------------------------------------------
And then the core/middle uses that to modify the dynamic batch file or
database. Here's the 'document' node:
----------------------------------------------------
<loci type='document' >
<widget>file_select</widget>
<input xml:link='simple' href='baselocus.xml' >
</input>
<output xml:link='simple' href='workspace1/viewer1.xml' >
</output>
</loci>
----------------------------------------------------
and here is the 'viewer' node:
----------------------------------------------------
<loci type='viewer' >
<widget>text_box</widget>
<input xml:link='simple' href='workspace1/document1.xml' >
</input>
</loci>
----------------------------------------------------
I think there's a lot we can collaborate on regarding XML.
> As for 1) I'm sure
> you can already do it with what you have. Simply edit an Overflow document that
> takes its inputs from a file and sends its output to another file (or stdout)
> and you're done.
I guess what we end up doing depends upon how much you and Dominic want to
change in Overflow. As for 1), this requires the least amount of work for you
guys. But if you're willing to take on new features and paradigms, I'm
thrilled to hear it :-)
> I forgot... another thing we have in Overflow that's essential (well, for what I
> do with it) is the ability to "automatically" cache the results that will be
> reused in another iteration of a loop or by another node, so nothing gets
> computed twice.
I haven't even thought about that, since we haven't developed any flow control
nodes for Loci. Loci does, however, always keep reference to the location of
a block of data (often kept as 'document' nodes) so we probably wouldn't have
thrown that stuff away.
Speaking of 'references', an important consideration we made when designing
Loci is that it's important not to transfer everything everytime. To solve
this, we keep a URI to programs, etc. and an XML REPRESENTATION of these
things. In most cases, only the representation needs to be transfered to the
local computer.
> I don't know whether that was your original goal, but what we're about to do by
> merging the 3 proejcts is this: a graphical shell (we could call it vsh, as in
> visual shell). Though with Overflow, it would also be a scientific application
> (the original Overflow goal was to be a "visual Matlab").
I like the sound of that. You know, Loci uses icons to represent nodes, and
some of the icons (will) look like those seen on a GUI desktop: folders,
trash, documents, etc. And we're following the Xerox/Apple model in many
cases: Drag-and-drop a node from a folder onto the desktop, double-click on an
icon to open a windowlet (little window that stays with the icon and in Loci),
etc. PLUS WE HAVE A PLAN FOR GRAPHICAL THEMES: The user can choose desktop
colors (and wallpaper when the canvas gets that item), icon styles, symbol
(goes under the icon to represent node class) styles that can be
mixed-and-matched with icons, plus Gtk themes that can be applied to Loci
widgets alone. The way I planned it, Loci COULD substitute for a user's
desktop.
As for a scientific application, Brad, myself and many others involved with
Loci are computational biologists who hope to use this for scientific
research. Hmmmm....There really are many, many more ideas that we've had, but
we'll get to those in time :-)
Cheers.
Jeff
--
+----------------------------------+
| J.W. Bizzaro |
| |
| http://bioinformatics.org/~jeff/ |
| |
| BIOINFORMATICS.ORG |
| The Open Lab |
| |
| http://bioinformatics.org/ |
+----------------------------------+
More information about the Pipet-Devel
mailing list