[Pipet Devel] IRC log about general DL/BL/PL design

[Brad Chapman]

Hey all;
	The IRC chat today didn't happen as plan due to Jarl's sickness
:-<. So there wasn't any time to do any arguing about design. Instead, I
asked a lot of questions of Jean-Marc about the design to get myself more
in touch with what he was thinking. 

The biggest change since yesterday is the idea of a Grandma-BL. This is a
since high level BL process which communicates with the multiple
mother-BLs. Jean-Marc and I talked about possibly implementing this in
python in the DL and thus changing the interface that the DL uses to
communicate with the BL. 

Anyways, the log is below for your perusal, if interested. Comments are,
as always, quite welcome.

Brad

[Snip...talking about Jarl being sick...]

<brad>	I'm coding the C++ CORBA stuff for the DL/BL communication. Fun
with C++ :-)
<jm>	BTW, any idea what's the best way to discuss about this splitting
thing with everybody?
<brad>	Well, here is an idea. Why don't you write something about it on
the piper list...
<jm>	Well, if we have a grandma-BL, there'll be no DL/BL Corba
communication. The grandma-BL will be linked against the DL and the CORBA
communication would be between the grandma-BL and the mother-BL's
<brad>	then start a page on the recently set up Piper Wiki
(http://www.bioinformatics.org/piperwiki/)...
<jm>	I can do that.
<brad>	then we could discuss it there and people could go through it. I
could set up a page for it.
<brad>	Then we'll have some kind of ready-made documentation on it -- or
at least something to start with...
<brad>	grandma-BL <- So you think the GBL will be in python? I'm
confused...
<jm>	Can't the GBL be done just like libflowui: a C/C++ dynamic
library, that the python DL could "use"/talk to?
<brad>	I guess so, although I really prefer CORBA over simply wrapping
things, since this helps keep some language independence.
<brad>	Additionally, I'm not sure how I would know if the GBL dies. With
CORBA it shouldn't be too difficult to know it died and restart, but I
don't know how this will work if I just wrap it.
<jm>	Well, if it's linked against the DL, then when it dies, the DL
dies too! Everything would be exactly the same way you use libflowui.
<jm>	Plus, I don't see why use CORBA when you are on the same
machine...
<jm>	...If you prefer, then you could write the GBL in python and put
it in the DL (the GBL would become the DL)
<brad>	dying <- Well, that is really bad, right? IF the DL dies then the
UI will just be left hanging. I'd like the DL never to die...
<jm>	There's no way you could be sure the DL won't die... except if
you don't run a DL at all :-)
<brad>	CORBA <- Well, I like CORBA a lot more than wrapping directly. I
don't think the slowdown over direct wrapping is that big of a deal, but
CORBA provides a nice interface to work off of, and would also make it
possible to rewrite the DL (if someone wanted to) in any language (ie.
java, if someone digs that).
<brad>	die <- Well, of course :-). But I don't want to the DL to die
just because the BL dies during a run! With a CORBA interface we should
be able to detect the death, and restart the BL as a separate process.
<jm>	Or what would you think about saying that the GBL and DL is one
thing? All written in python. Then all the MBL would communicate with the
DL.
<brad>	GBL in python <- Well, I don't know how much of the GBL
functionality Jarl already has in GMS. But I wouldn't mind writing it in
python at all.
<brad>	This is all provided that what Jarl has right now is mostly a
bunch of MBLs.
<brad>	Do you have any kind of vision for what the GBL/MBL interface
would look like?
<jm>	Sounds fine... but I doubt Jarl wrote the GBL, since we just
"decided" there would be a GBL. The functionnalities are for
viewers/probes and (maybe) splitting, none of which Jarl started
coding...
<jm>	Not at all. I have never touched (or even approached) CORBA... 
<jm>	I have an idea about the BBL/PL interface, but that's all! 
<brad>	The GBL in python sounds great by me (one more reason why the dl
is important :-).
<jm>	I have a slight idea of what the communications between the PL
and the DL should look like with no idea how it will get through the BL.
<brad>	Well, the BBL to PL interface is definately a good start :-)
<jm>	...and believe me, it won't be CORBA!
<brad>	I'd like to kind of see if we could at least hash out the rough
ideas of how the communication will look. I'm kind of at a loss about how
to proceed next with this, although I'm ready to code...
<brad>	no CORBA <- :-). No problems. I like CORBA a lot, but that
doesn't mean everyone has to...
<jm>	...not that I don't like CORBA (I don't know it enough to like it
or not). I just see the BBL linked to the PL.
<brad>	Yeah, I agree, this is also I have been thinking about it thus
far.
<jm>	I think the communication will be message/command based. The DL
will be able to send a PL node commands like "Next", "Continue", "Stop",
..
<brad>	But I have also been thinking about the BL/DL communication (or
GBL to BBL) as CORBA.
<jm>	A PL node will the the DL: "plot this data", ...
<brad>	The message based system makes a lot of sense with what we were
talking about in terms of that iostream stuff and having the error
logging system.
<jm>	Well, the GBL will not talk directly to the BBL's, there's MBL
watching ;-)
<brad>	I don't have a very concrete idea of how the BBL/PL stuff will
look, but what you say makes sense.
<brad>	MBL <- :-) Whoops sorry, I meant GBL to MBL. Still getting the
new acronyms straight.
<jm>	the BBL will just call getOutput() on its subnetwork. If at some
point a PL node needs to do a getOutput() on a remove machine, it will
ask the BBL for that data. How the BBL gets it is not the PL's problem.
In fact, it will end up as another BBL calling getOutput on its
subnetwork.
<jm>	...oops... I can't help on that!
<brad>	Okay, but how will the PL callback to the BBL to let it know it
needs info from a remote machine?
<jm>	let's say we have A -> (B->C) -> D, where B->C is on a certain
machine...
<brad>	Okay, I recognize this example :-)
<jm>	When breaking the network, B's input will be connected to a
special communication node. This node will call a BBL function to get
this data. the BBL will manage to get the data from A by talking
(indirectly) to another BBL and will return it to the special node, which
will return it to node B.
<brad>	Okay, so everything happens through the special communication
node. Is your idea for the design of this similar to the interface we
were kind of talking about with the iostream overriding class?
<jm>	no, this node will just cal a function. The only reason for the
iostream class is that I would like to be able to make that stream equal
to cerr (stderr) if I like without changing the code in the nodes.
<brad>	Okay, I'm with you. What do you imagine the function calls the
node will make looking like? Would this be similar to the interface type
that I sort of set up in the IDL file I sent a while back, or something
completely different?
<jm>	It would be sonething like: theBBL->getMeMyInput(this, inputID,
count);
<brad>	Okay, gotcha. BTW, what exactly is count? I saw it in all the
getOutputs() but I never exactly got it.
<jm>	("this" is a pointer to the current node, in case you're not
familiar with this part of C++)
<brad>	Okay what about errors generated during processing a getOutput()
in a node. How would those get returned to the BBL?
<brad>	this <- Yup. I'm with you on that :-) Thanks, tho!
<jm>	There's a count when loops are involved, it's used to select to
output number.
<jm>	it depends on what kind of error...
<brad>	Okay, so count is only relevant when in a loop.
<brad>	error <- Well, what different kinds do you imagine?
<jm>	some errors are "fatal": they throw an exception which will be
caught my the BBL. When there's an exception, it means that the task
simply cannot be performed (eg. a parameter is missing to a certain
node).
<brad>	Okay, that makes sense. What about non-fatal errors?
<jm>	Warnings through the streams (cerr << "something strange here" <<
endl;) will be sent to the DL (throught the BBL) and everything will
continue.
<jm>	when not in a loop, the count asked will be zero;
<brad>	What about an error which might cause the user to want to choose
between stopping and continuing (some kind of processing error)?
<jm>	The best example would be when treating video... if a node asks
another getOutput(outputID, n), it means I want to have the n'th frame. 
<brad>	Or how about your debugging/probe stuff?
<jm>	We will need to define a special interface for those... but I
haven't found a use for that so far (though not saying it doesn't exist)
<jm>	debugging will be handled through display/next/continue commands
<brad>	Okay, but we should just stick with simple cases first and see
when the other cases come up, you're right.
<brad>	I think this makes sense how the BBL/PL will happen. I guess we
need to talk through everything with other people to see how the GBL/MBL
should look...
<jm>	basically it'll be about commands (sometimes with data associated
to it)... The address of a node will me something like MBL:BBL:ptr  (the
address of the MBL, the address of the BBL, the (C++) pointer to the
node)
<brad>	Have you had a look at the dl2bl.idl interface Jarl and I wrote?
<jm>	nope... I know nothing about idl stuff, so I couldn't comment on
it anyway.
<brad>	commands <- I think (maybe) I am with you on this. I basically
understand the concept, although the implementation is foggy in my head.
<brad>	idl <- Well, it is very similar to C++ syntax, so it should be
hard to follow. I just thought maybe this might give you ideas about the
interfaces (ie. classes) we were thinking about for communicating between
the DL and PL. This was written a while back, but I think parts may still
be relevant to where we are at now.
<brad>	Ooops, that's 'shouldn't be too hard to follow' :-)
<jm>	Don't worry, the implementation is foggy for me too!
<jm>	I think the BL shouldn't need to know about the messages... The
BL will just transmit "data"
<brad>	implementation -> Well, as we go, can we sketch it out in C++
header file syntax or pseudocode or something? This would help me follow
a lot better, since I have some trouble thinking really abstractly :-).
We could use the wiki for this, again.
<brad>	messages <- I'm a bit confused about these messages all of a
sudden. Are these messages utilized by the PL?
<jm>	Well, there's no "detail" right now, the details will depend on
the type of node (viewer/probe/...)
<jm>	When the DL wants to say "Next" to a PL probe, it sends the
command through the BL, tough the BL never knows what "Next" means.
<brad>	Okay, so these are messages for DL to PL communication (and vice
versa). I'm with you.
<brad>	Will these messages be defined in some kind of XML syntax, in
your mind?
<jm>	Probably not... It will probably look a bit like a
pseudo-packet...
<jm>	There will ba an address (MBL:BBL:ptr), a command and some data.
<jm>	The data will be a serialized Object (deriving from my Object
class)
<brad>	Okay, but the only think I don't understand is how the ptr (the
most important part) will be transmitted about, especially between python
and C++.
<jm>	Or maybe, the command and data will be packed in an Object, which
will be the data in the pseudo-packet.
<brad>	serialized <- In what way?
<jm>	Well, a DL->PL communication will always be some kind of answer
to a PL->DL communication. The talking node will tell the DL its pointer.
The DL won't know what to do with it, but the BBL, which is in the same
address space (because it links to the PL) will.
<jm>	serialized: The whole Object (sometimes containing other Objects)
will be changed in a string of bytes.
<brad>	serialized <- This is my sticking point since I don't really
understand serialization :-< Sorry, I'll have to try and bone up on it.
<jm>	So far, I only support ASCII serialization: serializing a vector
of float looks like <Vector 1 2 3 7 2 4 6 1>
<jm>	If I have a vector of vectors, it will look like <Vector <<Vector
<1 3 5>> <Vector <4 5 6>>>
<jm>	The idea is just that an Object can be sent through a network or
saved to a file.
<brad>	Okay, so this is just C++ specific serialization right? Maybe I
do understand, since python has a pickle module which provides
persistance like this (ie. you can 'pickle' a data structure, and then
load it up from another program instance).
<brad>	I just don't understand it in C++ yet, but I get the idea now!
<jm>	It's not a C++ feature. it's just what you have to do to transmit
an object... maybe python does it for you, but it still needs to do it.
<brad>	Right, but there is a specific syntax that you need to spit it
out in so C++ can reload it (or transmit it across a network). What I
mean is that I don't yet know this syntax, but I can understand what is
it doing :-). How do pointers get serialized via this mechanism?
<jm>	I decided on the syntax... I wrote the save and the load. It's
like for a UIDocument that has contains a bunch of UINetworks that in
turn contain UILinks and UINodes, which contain UINodeParameters... You
manage to transform that to a bunch of bytes, which to save (the .n
files)
<jm>	For my Objects, instead of using libXML to serialize, I wrote my
own serializer, which is cleaner for what I do.
<brad>	Okay, understood completely. Sorry to be so confused on that!
<brad>	I think I sort of see where we are at, now we just need to move
forward and figure out the implementation stuff and see what other people
think about things...
<brad>	but that is probably for another chat session :-)
<jm>	Yep! In the mean time, I'll try to expose my ideas on splitting
on the list.
<brad>	Makes sense. I've been logging this, so I'll send it to the list
as well. Thanks for talking this over with me!
<jm>	Ttyl.
<brad>	ttyl. Have a nice night!