Hey all; The IRC chat today didn't happen as plan due to Jarl's sickness :-<. So there wasn't any time to do any arguing about design. Instead, I asked a lot of questions of Jean-Marc about the design to get myself more in touch with what he was thinking. The biggest change since yesterday is the idea of a Grandma-BL. This is a since high level BL process which communicates with the multiple mother-BLs. Jean-Marc and I talked about possibly implementing this in python in the DL and thus changing the interface that the DL uses to communicate with the BL. Anyways, the log is below for your perusal, if interested. Comments are, as always, quite welcome. Brad [Snip...talking about Jarl being sick...] <brad> I'm coding the C++ CORBA stuff for the DL/BL communication. Fun with C++ :-) <jm> BTW, any idea what's the best way to discuss about this splitting thing with everybody? <brad> Well, here is an idea. Why don't you write something about it on the piper list... <jm> Well, if we have a grandma-BL, there'll be no DL/BL Corba communication. The grandma-BL will be linked against the DL and the CORBA communication would be between the grandma-BL and the mother-BL's <brad> then start a page on the recently set up Piper Wiki (http://www.bioinformatics.org/piperwiki/)... <jm> I can do that. <brad> then we could discuss it there and people could go through it. I could set up a page for it. <brad> Then we'll have some kind of ready-made documentation on it -- or at least something to start with... <brad> grandma-BL <- So you think the GBL will be in python? I'm confused... <jm> Can't the GBL be done just like libflowui: a C/C++ dynamic library, that the python DL could "use"/talk to? <brad> I guess so, although I really prefer CORBA over simply wrapping things, since this helps keep some language independence. <brad> Additionally, I'm not sure how I would know if the GBL dies. With CORBA it shouldn't be too difficult to know it died and restart, but I don't know how this will work if I just wrap it. <jm> Well, if it's linked against the DL, then when it dies, the DL dies too! Everything would be exactly the same way you use libflowui. <jm> Plus, I don't see why use CORBA when you are on the same machine... <jm> ...If you prefer, then you could write the GBL in python and put it in the DL (the GBL would become the DL) <brad> dying <- Well, that is really bad, right? IF the DL dies then the UI will just be left hanging. I'd like the DL never to die... <jm> There's no way you could be sure the DL won't die... except if you don't run a DL at all :-) <brad> CORBA <- Well, I like CORBA a lot more than wrapping directly. I don't think the slowdown over direct wrapping is that big of a deal, but CORBA provides a nice interface to work off of, and would also make it possible to rewrite the DL (if someone wanted to) in any language (ie. java, if someone digs that). <brad> die <- Well, of course :-). But I don't want to the DL to die just because the BL dies during a run! With a CORBA interface we should be able to detect the death, and restart the BL as a separate process. <jm> Or what would you think about saying that the GBL and DL is one thing? All written in python. Then all the MBL would communicate with the DL. <brad> GBL in python <- Well, I don't know how much of the GBL functionality Jarl already has in GMS. But I wouldn't mind writing it in python at all. <brad> This is all provided that what Jarl has right now is mostly a bunch of MBLs. <brad> Do you have any kind of vision for what the GBL/MBL interface would look like? <jm> Sounds fine... but I doubt Jarl wrote the GBL, since we just "decided" there would be a GBL. The functionnalities are for viewers/probes and (maybe) splitting, none of which Jarl started coding... <jm> Not at all. I have never touched (or even approached) CORBA... <jm> I have an idea about the BBL/PL interface, but that's all! <brad> The GBL in python sounds great by me (one more reason why the dl is important :-). <jm> I have a slight idea of what the communications between the PL and the DL should look like with no idea how it will get through the BL. <brad> Well, the BBL to PL interface is definately a good start :-) <jm> ...and believe me, it won't be CORBA! <brad> I'd like to kind of see if we could at least hash out the rough ideas of how the communication will look. I'm kind of at a loss about how to proceed next with this, although I'm ready to code... <brad> no CORBA <- :-). No problems. I like CORBA a lot, but that doesn't mean everyone has to... <jm> ...not that I don't like CORBA (I don't know it enough to like it or not). I just see the BBL linked to the PL. <brad> Yeah, I agree, this is also I have been thinking about it thus far. <jm> I think the communication will be message/command based. The DL will be able to send a PL node commands like "Next", "Continue", "Stop", .. <brad> But I have also been thinking about the BL/DL communication (or GBL to BBL) as CORBA. <jm> A PL node will the the DL: "plot this data", ... <brad> The message based system makes a lot of sense with what we were talking about in terms of that iostream stuff and having the error logging system. <jm> Well, the GBL will not talk directly to the BBL's, there's MBL watching ;-) <brad> I don't have a very concrete idea of how the BBL/PL stuff will look, but what you say makes sense. <brad> MBL <- :-) Whoops sorry, I meant GBL to MBL. Still getting the new acronyms straight. <jm> the BBL will just call getOutput() on its subnetwork. If at some point a PL node needs to do a getOutput() on a remove machine, it will ask the BBL for that data. How the BBL gets it is not the PL's problem. In fact, it will end up as another BBL calling getOutput on its subnetwork. <jm> ...oops... I can't help on that! <brad> Okay, but how will the PL callback to the BBL to let it know it needs info from a remote machine? <jm> let's say we have A -> (B->C) -> D, where B->C is on a certain machine... <brad> Okay, I recognize this example :-) <jm> When breaking the network, B's input will be connected to a special communication node. This node will call a BBL function to get this data. the BBL will manage to get the data from A by talking (indirectly) to another BBL and will return it to the special node, which will return it to node B. <brad> Okay, so everything happens through the special communication node. Is your idea for the design of this similar to the interface we were kind of talking about with the iostream overriding class? <jm> no, this node will just cal a function. The only reason for the iostream class is that I would like to be able to make that stream equal to cerr (stderr) if I like without changing the code in the nodes. <brad> Okay, I'm with you. What do you imagine the function calls the node will make looking like? Would this be similar to the interface type that I sort of set up in the IDL file I sent a while back, or something completely different? <jm> It would be sonething like: theBBL->getMeMyInput(this, inputID, count); <brad> Okay, gotcha. BTW, what exactly is count? I saw it in all the getOutputs() but I never exactly got it. <jm> ("this" is a pointer to the current node, in case you're not familiar with this part of C++) <brad> Okay what about errors generated during processing a getOutput() in a node. How would those get returned to the BBL? <brad> this <- Yup. I'm with you on that :-) Thanks, tho! <jm> There's a count when loops are involved, it's used to select to output number. <jm> it depends on what kind of error... <brad> Okay, so count is only relevant when in a loop. <brad> error <- Well, what different kinds do you imagine? <jm> some errors are "fatal": they throw an exception which will be caught my the BBL. When there's an exception, it means that the task simply cannot be performed (eg. a parameter is missing to a certain node). <brad> Okay, that makes sense. What about non-fatal errors? <jm> Warnings through the streams (cerr << "something strange here" << endl;) will be sent to the DL (throught the BBL) and everything will continue. <jm> when not in a loop, the count asked will be zero; <brad> What about an error which might cause the user to want to choose between stopping and continuing (some kind of processing error)? <jm> The best example would be when treating video... if a node asks another getOutput(outputID, n), it means I want to have the n'th frame. <brad> Or how about your debugging/probe stuff? <jm> We will need to define a special interface for those... but I haven't found a use for that so far (though not saying it doesn't exist) <jm> debugging will be handled through display/next/continue commands <brad> Okay, but we should just stick with simple cases first and see when the other cases come up, you're right. <brad> I think this makes sense how the BBL/PL will happen. I guess we need to talk through everything with other people to see how the GBL/MBL should look... <jm> basically it'll be about commands (sometimes with data associated to it)... The address of a node will me something like MBL:BBL:ptr (the address of the MBL, the address of the BBL, the (C++) pointer to the node) <brad> Have you had a look at the dl2bl.idl interface Jarl and I wrote? <jm> nope... I know nothing about idl stuff, so I couldn't comment on it anyway. <brad> commands <- I think (maybe) I am with you on this. I basically understand the concept, although the implementation is foggy in my head. <brad> idl <- Well, it is very similar to C++ syntax, so it should be hard to follow. I just thought maybe this might give you ideas about the interfaces (ie. classes) we were thinking about for communicating between the DL and PL. This was written a while back, but I think parts may still be relevant to where we are at now. <brad> Ooops, that's 'shouldn't be too hard to follow' :-) <jm> Don't worry, the implementation is foggy for me too! <jm> I think the BL shouldn't need to know about the messages... The BL will just transmit "data" <brad> implementation -> Well, as we go, can we sketch it out in C++ header file syntax or pseudocode or something? This would help me follow a lot better, since I have some trouble thinking really abstractly :-). We could use the wiki for this, again. <brad> messages <- I'm a bit confused about these messages all of a sudden. Are these messages utilized by the PL? <jm> Well, there's no "detail" right now, the details will depend on the type of node (viewer/probe/...) <jm> When the DL wants to say "Next" to a PL probe, it sends the command through the BL, tough the BL never knows what "Next" means. <brad> Okay, so these are messages for DL to PL communication (and vice versa). I'm with you. <brad> Will these messages be defined in some kind of XML syntax, in your mind? <jm> Probably not... It will probably look a bit like a pseudo-packet... <jm> There will ba an address (MBL:BBL:ptr), a command and some data. <jm> The data will be a serialized Object (deriving from my Object class) <brad> Okay, but the only think I don't understand is how the ptr (the most important part) will be transmitted about, especially between python and C++. <jm> Or maybe, the command and data will be packed in an Object, which will be the data in the pseudo-packet. <brad> serialized <- In what way? <jm> Well, a DL->PL communication will always be some kind of answer to a PL->DL communication. The talking node will tell the DL its pointer. The DL won't know what to do with it, but the BBL, which is in the same address space (because it links to the PL) will. <jm> serialized: The whole Object (sometimes containing other Objects) will be changed in a string of bytes. <brad> serialized <- This is my sticking point since I don't really understand serialization :-< Sorry, I'll have to try and bone up on it. <jm> So far, I only support ASCII serialization: serializing a vector of float looks like <Vector 1 2 3 7 2 4 6 1> <jm> If I have a vector of vectors, it will look like <Vector <<Vector <1 3 5>> <Vector <4 5 6>>> <jm> The idea is just that an Object can be sent through a network or saved to a file. <brad> Okay, so this is just C++ specific serialization right? Maybe I do understand, since python has a pickle module which provides persistance like this (ie. you can 'pickle' a data structure, and then load it up from another program instance). <brad> I just don't understand it in C++ yet, but I get the idea now! <jm> It's not a C++ feature. it's just what you have to do to transmit an object... maybe python does it for you, but it still needs to do it. <brad> Right, but there is a specific syntax that you need to spit it out in so C++ can reload it (or transmit it across a network). What I mean is that I don't yet know this syntax, but I can understand what is it doing :-). How do pointers get serialized via this mechanism? <jm> I decided on the syntax... I wrote the save and the load. It's like for a UIDocument that has contains a bunch of UINetworks that in turn contain UILinks and UINodes, which contain UINodeParameters... You manage to transform that to a bunch of bytes, which to save (the .n files) <jm> For my Objects, instead of using libXML to serialize, I wrote my own serializer, which is cleaner for what I do. <brad> Okay, understood completely. Sorry to be so confused on that! <brad> I think I sort of see where we are at, now we just need to move forward and figure out the implementation stuff and see what other people think about things... <brad> but that is probably for another chat session :-) <jm> Yep! In the mean time, I'll try to expose my ideas on splitting on the list. <brad> Makes sense. I've been logging this, so I'll send it to the list as well. Thanks for talking this over with me! <jm> Ttyl. <brad> ttyl. Have a nice night!