Hey everyone; I've been on a mission to try and understand how Overflow works so that we can start to get it integrated into Piper. Towards this goal, Jean-Marc was kind enough to help me work through some of the areas where I was stuck. We thought the IRC conversation might also be useful for other people who are interested, so here goes. Enjoy! Brad brad-: Started <- Sure. Did what I was talking about in my e-mail make any sense? jm: Well, I'm having some problems with your terminology... brad-: What terminology? jm: But I understand you don't want to link the DL with Overflow brad-: Linking Dl and Overflow <- Well, I think most of the linking will be occuring at the level of the bl, right? brad-: If there are parts of Overflow you think would be useful in the dl, I'm definately more than happy to use them! jm: I don't understand what you're refering to by "A piper user should be able to wrap any program" brad-: Okay, what I'm trying to talk about is having a user be able to convert a program (or library) to be able to run under piper. jm: Well, all Overfow nodes need to be classes... using a program has to be done by wrapping it in a node. brad-: I've been trying to go through Overflow to see how the libraries you are using get loaded to work with the program, but I get kind of lost. I can't see the link... brad-: When you say wrapping in a node, how does this work. For instance, let's just take the HMM library. Where in the code does it get integrated into Overflow? jm: Well when you do a dlopen() on the librairy, the code for the node gets loaded and there's an initializer that sets some data about how to create each node in the lib, as well as what inputs/outputs/parameters are required brad-: Okay, this was the general picture I had. What specifically is the initializer that sets the data and figures out the inputs/outputs/parameters? jm: BTW, all those libraries contain classes that all derive from the base "Node" class jm: If you look at .cc files that define nodes, all have the NODE_INFO(...) macro at the start. jm: The macro arguments are the node name, category, inputs, outputs, parameters brad-: Oh, macros, okay. Where is the macro defined? jm: It's in data-flow/include/Node.h brad-: Okay. Gotcha on that. So If I wanted to get a new library working under Overflow, what steps would I need to take? jm: Basically, it uses a C++ initializer to call addFactory(...) with all the data. brad-: Yeah, I'm just looking at it. Man, I was having the hardest time trying to figure out how NodeFactories got added :-) jm: You create a class that derives from Node and understands the getOutput(...) method. Then you include the NODE_INFO macro and link with it brad-: Okay, makes sense, I gotcha. What are your thoughts on ways to move this system into piper? jm: The idea is that in C++, you can define a global variable as int dummy = some_funct(); and some_funct() will get called before the main starts (or when the library is linked) jm: Two ways: 1) get the data by linking with the library (which you don't like), or 2) get the data somewhere else. jm: One way would be to create an executable that loads all those libs and outputs the content of the NodeDictionary in an XML file. brad-: Get the data by linking with the library <- It is not that I don't like this :-). I just was trying to think of a way to generalize it so people wouldn't have to know c++ to get a program working under piper. jm: What do you mean by "get programs working under piper"? brad-: loading libs to XML <- This sounds very nice :-). That sounds like a good way to go. The Overflow could link in the libraries and get their info, and I could link in the libraries in the dl and output the info as an XML file. This way the info would then be available to send to the ui. jm: There is already a node (C++ class) that takes a program name as input and outputs the stdout of the program brad-: "get programs working under piper" <- Oh sorry, my bad terminology again :-). What I'm just trying to say: integrate a program into piper so that it can be used as a piper node. brad-: program name node <- Where is this defined? Also, how can we get at the available flags, etc to send to the program? jm: There will be a set of nodes that can "talk" to other programs... But no need to code a new node for each program you want to use in piper. jm: It's called ExecStream brad-: ExecStrem <- Gotcha. I'll talk a look at this later and understand what is going on :-) brad-: Okay, maybe I should stop chatting now and take some time to look things over again with all of the new info :-) jm: I'm going to have dinner anyway... ttyl brad-: I'll have better questions to ask you once I manage to understand things more. jm: BTW, we should save the log... it might help other people... brad-: Have a good dinner. Thanks for talking me through this, I really appreciate it! ttyl