From jeff at bioinformatics.org Fri Sep 1 21:35:19 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:26 2006 Subject: [Pipet Devel] [Fwd: Is Piper for real?] Message-ID: <39B05957.6E5B2C54@bioinformatics.org> This is an e-mail Gary and I (any others?) got today about Piper. Our responses follow. Jeff -------------- next part -------------- An embedded message was scrubbed... From: Cameron Laird Subject: Is Piper for real? Date: Fri, 1 Sep 2000 05:52:44 -0500 (CDT) Size: 1176 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000902/ba48937a/attachment.mht From jeff at bioinformatics.org Fri Sep 1 21:35:45 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:27 2006 Subject: [Pipet Devel] [Fwd: Re: Is Piper for real?] Message-ID: <39B05971.1800CB7F@bioinformatics.org> -------------- next part -------------- An embedded message was scrubbed... From: Gary Van Domselaar Subject: Re: Is Piper for real? Date: Fri, 1 Sep 2000 08:24:03 -0600 (MDT) Size: 2806 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000902/13468764/attachment.mht From jeff at bioinformatics.org Fri Sep 1 21:36:08 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:27 2006 Subject: [Pipet Devel] [Fwd: Re: Is Piper for real?] Message-ID: <39B05988.EEC91733@bioinformatics.org> -------------- next part -------------- An embedded message was scrubbed... From: "J.W. Bizzaro" Subject: [tol-advisors] Re: Is Piper for real? Date: Fri, 01 Sep 2000 22:42:35 +0000 Size: 10290 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000902/119a0fbc/attachment.mht From jeff at bioinformatics.org Mon Sep 4 12:06:05 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:27 2006 Subject: [Pipet Devel] [Fwd: Is Piper for real?] Message-ID: <39B3C86D.204F5DD8@bioinformatics.org> -------------- next part -------------- An embedded message was scrubbed... From: Cameron Laird Subject: Re: Is Piper for real? Date: Mon, 4 Sep 2000 11:02:24 -0500 (CDT) Size: 4422 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000904/01b28be3/attachment.mht From jeff at bioinformatics.org Mon Sep 4 12:23:21 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:27 2006 Subject: [Pipet Devel] [Fwd: Is Piper for real?] References: <39B3C86D.204F5DD8@bioinformatics.org> Message-ID: <39B3CC79.4AED2B41@bioinformatics.org> I wrote back to Cameron and told him that his description is accurate. If you have any comments, please mail them to Cameron and CC to the list. Cheers. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jeff at bioinformatics.org Mon Sep 4 12:38:04 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:27 2006 Subject: [Pipet Devel] BlueBox Message-ID: <39B3CFEC.6115D0B8@bioinformatics.org> Greetings. Before Brad and I found out about ISYS and OpenBSA, we planned on a collaboration with the developers of BlueBox (developed by dLoo Software and licensed under the LGPL): http://www.dloo.com/products/index.html I'm trying to get Nile Geisinger (head developer of BlueBox) on this list. In the meantime, I think you might want to take a close look at BlueBox. I believe OpenBSA, ISYS, and BlueBox have a great deal in common. Also, the thought has been brought up more than once (even with Nile) that Piper may be best used as a pipeline infrastructure for systems like BlueBox, OpenBSA, and ISYS, and that Piper should generally leave the user interface end of operations to those systems. I think this may be the working ground for DNP API specifications. Cheers. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From mangalam at home.com Mon Sep 4 11:52:57 2000 From: mangalam at home.com (Harry Mangalam) Date: Fri Feb 10 19:20:27 2006 Subject: [Pipet Devel] Re: [dnp-api] BlueBox References: <39B3CFEC.6115D0B8@bioinformatics.org> Message-ID: <39B3C559.FD8E96DF@home.com> Hi Jeff (and good to meet you at ISMB), It sounds pretty interesting, but just how do we find out anything substantive about it? There's nothing but a static page with sales gibberish with no useful links at the URL you supplied, and most of the google-derived web pages that might have pointed to something informative have been 404'ed.. Being a recent (re)subscriber I may have missed a previous substantial thread - Do you have a summary or pointers to what BB actually is? Harry "J.W. Bizzaro" wrote: > > Greetings. > > Before Brad and I found out about ISYS and OpenBSA, we planned on a > collaboration with the developers of BlueBox (developed by dLoo Software and > licensed under the LGPL): > > http://www.dloo.com/products/index.html > > I'm trying to get Nile Geisinger (head developer of BlueBox) on this list. In > the meantime, I think you might want to take a close look at BlueBox. I > believe OpenBSA, ISYS, and BlueBox have a great deal in common. > > Also, the thought has been brought up more than once (even with Nile) that > Piper may be best used as a pipeline infrastructure for systems like BlueBox, > OpenBSA, and ISYS, and that Piper should generally leave the user interface > end of operations to those systems. I think this may be the working ground > for DNP API specifications. > > Cheers. > Jeff > -- > J.W. Bizzaro jeff@bioinformatics.org > Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff > "Injustice anywhere is a threat to justice everywhere." > -- Martin Luther King, Jr. > -- > > _______________________________________________ > dnp-api maillist - dnp-api@bioinformatics.org > http://bioinformatics.org/mailman/listinfo/dnp-api -- Cheers, Harry Harry J Mangalam -- (949) 856 2847 (v&f) -- hjm@ncgr.org || mangalam@home.com From jeff at bioinformatics.org Mon Sep 4 13:18:26 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Re: BlueBox Message-ID: <39B3D962.B22BD7C0@bioinformatics.org> Harry Mangalam wrote: > > Hi Jeff (and good to meet you at ISMB), It was good to meet you too :-) > It sounds pretty interesting, but just how do we find out anything > substantive about it? There's nothing but a static page with sales > gibberish with no useful links at the URL you supplied, and most of the > google-derived web pages that might have pointed to something informative > have been 404'ed.. > > Being a recent (re)subscriber I may have missed a previous substantial > thread - Do you have a summary or pointers to what BB actually is? Well, how about that! They used to have lots of links to details and the source code. I found that dLoo still has some pages up about Piper and development plans: http://dloo.com/piper/index.html I also attached some e-mail messages that may be enlightening. But, as far as /real technical specifications/ are concerned, we haven't seen any. BlueBox has been in this pre-alpha/alpha stage, along with Piper, for some time. I hope this standardization project will bring out all the specs. Cheers. Jeff -------------- next part -------------- An embedded message was scrubbed... From: "J.W. Bizzaro" Subject: [Pipet Devel] dLoo BlueBox Date: Sun, 11 Jun 2000 04:52:34 +0000 Size: 4183 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000904/2235fd33/attachment.mht -------------- next part -------------- An embedded message was scrubbed... From: "J.W. Bizzaro" Subject: [Pipet Devel] [Fwd: GNU Piper] Date: Mon, 12 Jun 2000 02:25:40 +0000 Size: 9296 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000904/2235fd33/attachment-0001.mht -------------- next part -------------- An embedded message was scrubbed... From: "J.W. Bizzaro" Subject: [Pipet Devel] [Fwd: GNU Piper] Date: Tue, 13 Jun 2000 01:10:35 +0000 Size: 4358 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000904/2235fd33/attachment-0002.mht -------------- next part -------------- An embedded message was scrubbed... From: Nile Geisinger Subject: [Pipet Devel] Greetings from dLoo Date: Wed, 14 Jun 2000 02:35:14 +0100 Size: 3732 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000904/2235fd33/attachment-0003.mht -------------- next part -------------- An embedded message was scrubbed... From: Brad Chapman Subject: Re: [Pipet Devel] Greetings from dLoo + answers to questions Date: Thu, 15 Jun 2000 05:11:01 -0400 Size: 12101 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000904/2235fd33/attachment-0004.mht -------------- next part -------------- An embedded message was scrubbed... From: Nile Geisinger Subject: [Pipet Devel] Piper's Importance and Resources Date: Fri, 16 Jun 2000 12:23:13 +0100 Size: 3977 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000904/2235fd33/attachment-0005.mht -------------- next part -------------- An embedded message was scrubbed... From: Brad Chapman Subject: Re: [Pipet Devel] BlueBox source code Date: Tue, 25 Jul 2000 09:13:30 -0400 Size: 3215 Url: http://bioinformatics.org/pipermail/pipet-devel/attachments/20000904/2235fd33/attachment-0006.mht From jean-marc.valin at hermes.usherb.ca Mon Sep 4 13:33:45 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] BlueBox References: <39B3CFEC.6115D0B8@bioinformatics.org> Message-ID: <39B3DCF9.3BD10902@hermes.usherb.ca> > Also, the thought has been brought up more than once (even with Nile) that > Piper may be best used as a pipeline infrastructure for systems like BlueBox, > OpenBSA, and ISYS, and that Piper should generally leave the user interface > end of operations to those systems. I actually think Piper will require many user interfaces, since users will be doing things so different with it. For example, you can use Piper just to replace the pipes on the command-line (for non-technical users), while you can use it to do distributed parallel calculations for [signal processing/bioinformatics/artificial intelligence], or for fast real-time sound effect processing. I don't think one (or even a few) UI can fit all needs. Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jeff at bioinformatics.org Mon Sep 4 13:51:10 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Re: BlueBox References: <39B3CFEC.6115D0B8@bioinformatics.org> <39B3DCF9.3BD10902@hermes.usherb.ca> Message-ID: <39B3E10E.F07E101C@bioinformatics.org> Jean-Marc Valin wrote: > > I actually think Piper will require many user interfaces, since users will be > doing things so different with it. For example, you can use Piper just to > replace the pipes on the command-line (for non-technical users), while you can > use it to do distributed parallel calculations for [signal > processing/bioinformatics/artificial intelligence], or for fast real-time sound > effect processing. I don't think one (or even a few) UI can fit all needs. Very true, thank you. For those on the DNP-API list, you'll find more about Overflow (the native processing engine of Piper) at its website: http://freespeech.sourceforge.net/overflow.html Cheers. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jarl at casema.net Mon Sep 4 23:43:27 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Is this accurate? Message-ID: <39B46BDF.CA97AB9@casema.net> Hi Cameron, I'm a co-developer of Piper and the author of one the the projects namely GMS, or Generalized Messaging System. This name should be corrected in your text. Furthermore, I think one of the most important features Piper will offer is multiple path piping. On normal UNIX's you can pipe ("|") only from one source to another. Piper can merge or split streams. I think your article is great; it's easy to read and covers Piper very well, thank you for this. bye, jarl -- http://sunsite.auc.dk/gms From jean-marc.valin at hermes.usherb.ca Mon Sep 4 15:04:02 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Is this accurate? References: <39B46BDF.CA97AB9@casema.net> Message-ID: <39B3F222.4BF68C03@hermes.usherb.ca> > Workers in the life sciences seem to have the most enthusiasm for > Piper; genetics researchers, for instance, can easily see > the benefits of a system that makes it easy to build > filters and computations on a remote store for the human > genome or other massive datasets. I'd also like to add that not all Piper developers are working on Life sciences. I'm co-author of the Piper processing layer (the Overflow project), which was primarly developped for signal (speech, image) processing, from an electrical engineering point of view. I think Piper is more than a life science thing. Other than that, I agree with the article. Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jeff at bioinformatics.org Mon Sep 4 17:19:12 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Is this accurate? References: <39B46BDF.CA97AB9@casema.net> Message-ID: <39B411CF.D53E5304@bioinformatics.org> jarl van katwijk wrote: > > I'm a co-developer of Piper and the author of one the the projects namely GMS, > or Generalized Messaging System. You mean I've been spelling it wrong all this time? :-) Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jeff at bioinformatics.org Mon Sep 4 17:41:05 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Is this accurate? References: <39B46BDF.CA97AB9@casema.net> <39B3F222.4BF68C03@hermes.usherb.ca> Message-ID: <39B416F1.D70C16F6@bioinformatics.org> Jean-Marc Valin wrote: > > I'd also like to add that not all Piper developers are working on Life sciences. > I'm co-author of the Piper processing layer (the Overflow project), which was > primarly developped for signal (speech, image) processing, from an electrical > engineering point of view. I think Piper is more than a life science thing. I second that notion. I have presented Piper to my colleagues in the life sciences as a system that can be very useful to them, but I am always sure to mention that Piper is general-purpose in design. I believe that this makes Piper even better for special-purpose use, since it can change along with standards. It also attracts a broader mindshare of users and developers, which only helps to improve the system for everyone. Perhaps we are not all scientists, but all of the developers appreciate the need for a system capable of very sophisticated operation, yet with a simple facade. Cheers. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jarl at casema.net Tue Sep 5 11:34:08 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Is this accurate? References: <39B46BDF.CA97AB9@casema.net> <39B411CF.D53E5304@bioinformatics.org> Message-ID: <39B51270.3B321AB5@casema.net> > > I'm a co-developer of Piper and the author of one the the projects namely GMS, > > or Generalized Messaging System. > > You mean I've been spelling it wrong all this time? :-) Think you took the save way of using 'GMS' :) bye, jarl From chapmanb at arches.uga.edu Tue Sep 5 12:51:42 2000 From: chapmanb at arches.uga.edu (Brad Chapman) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] IRC on Bl/Pl organization Message-ID: <200009051651.MAA119342@archa10.cc.uga.edu> Hey all; Jean-Marc, Jarl and I have been trying to work on thinking up good plans for how to organize the BL and PL and get some good connectivity going. Yesterday there was a big ol' IRC chat about this, which is pasted below. It discusses the organization of the BL and how this will interact with the PL. This discussion introduces the ideas of central "mother BLs" (MBLs) which are central controlling BLs that will interact with the DL (but maybe not directly). The mother BL can then have "Baby BLs" (BBLs) which will directly interact with the PL. This is sort of a summary of what is going on, but the actual structure is still under heated discussion, as you can see from reading the log. Part 2 of this discussion will occur today at 4:30pm EST. Anyone who is interested is welcome to join in (or listen). The chats occur on ChatNet (see www.chatnet.org for a list of servers) on the #piper channel (naturally enough). See you there! Brad --> jm (~jm@cn-997.143-201-24.mtl.mc.videotron.net) has joined #piper hi jean-mark I'm here! Hi Jean-Marc! Have you been talking for a long time? got the email or the icq? e-mail... got no ICQ! almost an hour already :) icq sucks :( ..oops I actually had the ICQ! --- bstard gives channel operator status to jm talking <- mostly about compiling the BL and fun stuff like that :-) And how's the C++ compiling going? and I screatched a meganism for the work the BL should do Go ahead... C++ port <- went smoothly until I met the C++ Corba C++ corba is very diff. from C Corba. 0: DL logs into BL 1: DL uploads XML network to BL 3: BL checks XMS for resources 4: BL distributes nodes to BL instances agross internet 5: Acknowledges DL and PL that BL is ready and waiting 4 a go 6: {undefined yet} step 4 will be the one that will undergo big enchancements once we've the basics operational. stuff like resource management etc will be runtime some day in the future :) So wouldn't the BL involved in 3 and 4 be a "master" while the distributed ones would be "slave"? oops, forgot 2: BL checks XML for authorized nodes master-slave: ok, way of looking at thing. things like you view it: master = local instance, slave = remotely (on other machines) so a BL is master if it's distributing, and a slave once it's receaving XML's Sort of, what I meant by "master" (may not be the best term) is the part that has the "global view" while all the other BL instances ("slave") only see a small part of the problem. Yeah, that is the view I had of master/slave as well. But once the network is running, the BL instances are 'same level' instances communicating results etc to each other. The master-slave relation only exsists during the 'distribution phase' jm: jep, that's the way I view the situation too. No, but I think that things also need to be broken up so that you can return information back to the DL, like in the "viewer" nodes. slave = partially knowledge of the 'job', master has full knowledge Well, I think there should be a master that is the only one allowed to talk to the DL. brad: what do you mean by 'no' ? no <- I mean, doesn't the master-slave situation also need to exist on a local level? jm: the DL logs into a BL, so only that DL can receive, only that DL will be offered the results and can upload jobs Well, if one part of the processing is done locally, then there should be a slave running on the local machine. nah, why an extra thread\instance? the way I see things, only 1 BL will run on a machine. but try to convince me :) What I meant was that all when slaves want to communicate with the UI, they should all send the info the to master. Then the master is the only one allowed to talk to the DL. convince <- Jean-Marc is probably the best person to do this, since it is his idea :-) hehe I see the master as a library that's linked with the DL, so there's one process for the DL and the master. uhh, how do I put this. See the DL as the master BL, that;s probably the best way to look at the situation Also, it you have an SMP machine and want to run two part of the processing on it, you should have two BL slaves running on this machine. all the BL's do the work for THAT specific DL they also do work for other DL's jm: SMP will be covered by internal threading which is already implemented for 50%.. Why not just say one BL slave process for each part of the processing? multiple DLs <- Jarl, this is confusing me. There will only be one DL process per machine. A single process can handle multiple user interfaces. but.. I still can not see why you people advocate a master-slave model. Try me again please.. jm: each part of the processing will be done by the PL? So it seem more logical to thread multiple PL's brad: did I say "Multiple DL's" ?? If I did, a mistake, sorry Ofcourse we'll only need 1 DL/mache machine arf multiple dls <- Okay, sorry. I was just getting confused :-) I'm thinkint 1PL=1BL, so there's as much BLs as there are processing parts. brad: who aint :) jm : good point.. they are gonna be statically linked which kinda sucks regarding my views :-) What's going to be statically linked? Overlow as lib to the BL or will we use .so ? Well, it "has" to be dynamically linked ic why? Overflow is meant to work dynamically, since the toolboxes are dynamic libraries that are loaded on startup. ok, no problem and Overflow can only execute 1 network at a time? Basically, when Overflow starts, it looks for all the .so files in a certain path and loads them all. Each .so contains code for a bunch of nodes. That way, you can add nodes without recompiling. ic, that's the ways GMS works with plugins too Well, Overflow is just a library, so you could build 10 network and run them at the same time if you have 10 threads. And could you implement threading in the library after the RUN object? so that the BL could upload multiple XML's into the PL and give it a go ? Well, it would be equivalent to the BL starting multiple threads and starting a processing in each one. :) hehe so I'm buggered again? :-) dont get me wrong The reason I prefer one PL per BL is that sometimes the PL could crash for some reason and you want to be able to track it. Also, you could restart only this part instead of all the PL's running on that machine. I cant really oversee the situation because I dont really know how Overflow is implemented.. If you think the best way is to have the BL do the threading that's fine to me... crashing <- ic crash <- I think this is very important. People will really want to know why things crashed, and during development we will definately need to know this :-) dont expect the BL to be 100 stable.. it HAS been running for days under stress testing.. but I you never know "but I you never know" <- but you never know :) stable <- No problems, I don't expect anything to be stable yet :-) OK, so the design will be : 1 DL -> 1 BL with internal threading -> 1 PL / BL thread ?? I think in this case, mulpiple processes are better than multiple threads. You're losing me... aailks loosing where? @ the threading vrs. processes? I don't understand what you want to thread. In my mind we wouldn't need any thread at all (just separate processes). I still am in faivor of 1 BL process / machine.. that's why I'm thing about threading multiple BL's will be very hard to manage darn... if only we were sitting next to each other I could draw a design :) that would clear up thing\ s jm: you saw this flowchart on http://sunsite.auc.dk/gms ? The problem is that if the PL crashes, it will crash the BL and all the other PL's on the machine. hmm... I can have the BL run inside a debugger for as long as things are unstable a build-in debugger no gdb or so.. but the ONLY thing you have again a sigle process design is the instability? Or do you just dont feel confortable with it? it's more about PL instabilities and error recovery. hmm.,.. just a sec, I'll give somebody a call about we running the BL\PL set inside a debugger.. just to ask if it's possible. arf... if you need people :( no answer :) otherwise we can have the DL do the resurrecting? it's not only about debugging but also real-life. If the PL crashes in real-life, then you only need to restart one PL. the BL is capable of keeping track of a runtime history, so it can do a full restore after a crash no, not debugging the errors! DL resurrection <- I can detect a BL crash very easily, but I'm not sure how I'll be able to report back to the UI on what caused the crash... Not if the crashing PL crashes the BL and the other PL before they have time to terminate. a debuggers receives the crash signals and can respawn the BL\PL process but you loose all the computations from the other PL's DL resurrection <- the DL can restart the BL with a special parameter, and the BL will do the rest just like if we'd BL slaves restart a PL process. Dl resurrection <- What do you mean be special parameter? jm: lose other computations. again good point. like ./BL -restart :) ok, the PL can not do history tracking.. that would become too slow. shit! sorry I don't see what's so complicated about running two BL's on the same machine. ok. What about: the BL will start a child\slave process for all data streams coming out of the Pipeline? out of the BL pipeline that is. What do you mean by BL pipeline? all incoming data streams go thru the pipeline, that does the authentication etc. see : http://sunsite.auc.dk/gms/gms.gif other things like marshalling data and resource management are also located in the pipeline Sorry, I don't understand your flowchart... if we are to use multiple BL's on 1 machine that will become very hard look only at the middle part, with 'outside world' above it the 'sensors' are the data streams coming from external sources.. What's harder? For me it's just communicate with another BL, regardless of where it is (local or remote) the basic of gms is that there is only 1 pipeline on a machine. That's why I make the fuss about multiple BL's :-) so the plan of stripped (non-pipeline, no DL-communicating, non-external input) slave BL's sounds ok? or dont you get it? Sorry, I think I'm missing some basic information... to you ( the PL) there wont be a diff. Also to the DL there's no diff. and the BL design will remain intact jm: maybe we need to discus this on voice some day? brad: you're still here? :-) Well, you're the BL specialist after all... it's just that from my point of view, things would look so much cleaner if "one part of the processing" = 1 BL "slave" = 1 PL, regardless of where everything runs. Yup, just listening :-). This is your guys battle. ...Just thinking... is the multiple BL problem about the authentication and stuff like that? jep. about deadlocks and so on. and about resource management In this case, maybe there could be a permanent part of the BL on each machine, which would spawn (process) Baby-BL's linked with the PL. that's why I only wnat to have 1 process doing all that on 1 machine YEP! the BL ' master' will be permanent those 'slaves' will be spawned one there's something to do 1 spawned stripped-down BL / PL once? and die when the job is done That's what I'm thinking about... it's just that I had forgotten about all that authentication/resource thing. Can you wait a sec... ok, pfeeew we've settled something it seems :) Good stuff! Baby-BLs are the way to go :-) ofcourse I'll wait... /me having a thinking-smoke brad: lol hehe BBL's :) :-) I'm happy you are coming together on that. The Baby-BLs actually makes a lot of sense with what both of you were saying. Maybe we can have parent/baby relationship, instead of master/slave. Sounds a little nicer :-) brad: could you build into the DL a respawn check for the Mother BL ? the MBL so to say What do you mean by a respawn check? if the MBL dies, the DL should restart the thing like the BBL's do for the PL' s Yeah, that is no problem. We should add a little ping() function to the IDL, then when the ping fails, the DL will restart the BL. Well the BBL would link to the PL, so both die at the same time it would be the mother BL that would respawn a BBL/PL brad: ping <- yep, I'll put it into the IDL jm: jep! So what is our plan for actually implementing all of this? the MBL will do all the resource\AAA and (re)spawning work As for what I was calling the "master", it could belong to the DL or BL (I don't care), and that's the one I see responsible for splitting the processing, sending that to the MBL's and respawning MBL's Actual impl. <- 1stly the C++ Corba should be done. After that I'll be eager to do some coding of new features :) jm: I see what you mean now. I think you can see my initial opposition to this now too can you? opposition to what? to the master-slace design slave impl <- Well, how about if we split like this: I'll work on getting the CORBA C++ working and you and Jean-Marc can work on implementing this whole mother/baby thing.... if slave start to do authentication etc the system will become a big mess I'm not sure you understand what I mean. What I was calling "master" is more like a library that the DL would use to communicate with all the mother-BL's brad: I can not devel that without the sources compiling :( that darn corba must be running soon For me master BL != mother BL. jm: ?? master != mother? OK, let me say how I now see the whole thing: impl <- Well, lets extract the CORBA out into its own separate module. Can you just write a little C++ code that will run to test things, and then we'll make things be driven by the CORBA subsequent to that? I think the CORBA should come out of the messaging_db stuff for cleanliness maybe anyways... The DL links (as library) with the "master"-BL, which takes care of splitting the processing into small XML chunks that are sent to all the mother-BL's on the different machines. Then each mother-BL spawns a Baby-BL/PL for each part it receives. brad: let's not create too much chaosshall we? shall we 1st finish the corba, and once it's working I'll separate the corba and the database, ok? jm: three BL's? hmm Assuming the flow A->B->C where A and C are run on machine X, B is run on machine Y, and the DL is run on machine Z, here are the processes I see: 1 process on Z (the DL/master-BL) 3 processes on X (1 Mother-BL, 2 baby-BL's) 2 processes on Y (1 mother-BL, one baby-BL) Once again, the confusion probably arises from the fact that you probably don't consider my "master-BL" to be part of the BL... impl <- I'll put this on hold until you and Jean-Marc are done... brad: k :) jm: ic what you mean I see this master BL and mother BL as being the same process the motherBL gets a chunk of XML from the DL, splits it, sends it to other motherBL's and the motherBL's start producing baby's I see all the mother-BL's being equal -> no reason why the splitting would be done by one particular instance of mother-BL. and maybe the 'other' motherBL's will also split the XML once more if they see fit hehe jm: still see the need for a 3th BL type, the master BL? ...or you can call it master-DL if you like, since I don't see where it belongs. I think it's not the mother-BL's job to split the processing in chunks. ...if you don't do processing on the local (where the DL runs) machine, you don't need a mother-BL there. but the BL will manage the resource management, this can not be done without doing the xml splitting Sure, but since there are many BL's, the splitting decision has to be split somewhere. There should be a part that only does that. there will be only 1 MBL on a machine. So it's capable of desiding the splitting why introduce a new process for just doing the splitting? the MBL will do 1: authenticate, 2: recouremanagement\splitting 3: spawning BBL's Yes, but how do you decide how to split between different machines. What I'm saying is that there should be a part that's in the middle of the DL and all the mother-BL's there are many MBL's, right? one on each machine I think a MBL should "take what it want" and "pass on what it doesn't like to other MBL's" no, just ONE MBL / machine many BBL\PL sets ...OK, let's go from the start. ok :-) If we have 10 machines, we have 10 MBL's (one on each machine). Let's assume the DL runs on a 11'th machine. ok.. The DL gets from the GUI a complete (unsplit XML network). (remember, there's no MBL running where the DL is) There's no reason the DL would ask one particular (of the 10) BL's to do the splitting. there is the DL doesn't have login access to every MBL but the DL COULD have it.. very strange situation though I think there should be "something" on the local machine that (maybe after communicating with all the MBL's) decides how the program should be split and then sends each part to the right MBL. Why not have a special part of the BL that would do the interface between the DL and all the MBL's (on all the machines) what about making this "something" this logic: "take what it want" and "pass on what it doesn't like to other MBL's" that way the splitting can start on every system on the net in theory.. I think we disagree on how the processing will be split... From what I understand, you say that only the BL will decide about the splitting. Am I right? you say splitting should be done by a splitting master, I think we can rely upon smart algorithms that will do the job without any 'mastering' I'm more thinking that the user will say "This runs here, that runs there, ..." the user CAN say that Only, sometimes, the user want something to run somewhere... and nobody's going to decide something else. that way the splitting will be very easy but what if the user says 'just runs it somewhere where it will be done fastest' ? In this case the local "splitter" would ask all the MBL's about their resources, but I think most of the time the decision will not be made by the MBL's. Let's say I have a sound acquisition node, I want it to record from a certain microphone that's on a certain machine... not to record on any microphone, where the BL like to. I think that way will build a very static system. You should have more faith in having the MBL's fighting about getting the optimal resource load microphone: ok, in that case you define to use THAT specific mic. Splitting highly defined xml will be easy, we'll just pass the xml on to where the user wants it to be I'm more interested into xml that does NOT contain any localisation defines I think the user needs to have a word to say about the splitting. sure he\she does! the user work with 'that file' and 'this soundcard' but also with 'some computation power, where the heck it might be' or 'information about this' What not have all the MBL's send their resource info at the same place and then combine that with what the user likes and take the decision *in one place*. how will this be done once Piper runs on 103430943 machine some day in the future? machineS arf There are always things the user will know that the BL won't know... like "if I run this part that lauches this process on this machine, than this user will be mad at me because of this side effect". There's no way the BL can handle that! ok, that's why the MBL will get strict resource limits maxcpu = xx% maxmem = xx mb etc but ok, you're a tiny bit right :) sometimes, it's just that a certain node needs more io or does something else that disturbs... the BL wont know everything.. so wont the 'splitter BL' Yes, but the user can speak to the splitter BL and help it do the splitting. that's why I want to implement a 'prediction' neural net once we've Piper running.. to predict availeble resources hmm.. you think the user should guide the splitting? I think the user can define anything he wants, after which it's up to the BL to fill in the gaps Anyway, I don't see how distributing the splitting would be better than making a centralized decision taken from the data (resources) of all the BL's. Note that I don't see how the BL will know what resources (CPUt time) are going to be required for each node, since it will not know what the nodes do. 1st because a centralized meganism wont be possible once there are a lot of Piper's running, and 2ndly because I know prediction algoritms are much more effective than centralized algoritms When I write new nodes, I don't want to have to tell the BL how long it will take to run, depending on all the variables (size of the problem, ...). not know how much it will use in advance <- ok, not the 1st time the node runs, but here the history logs and prediction algoritms come into play but I hope you'll reuse those nodes a next time you build a network so the MBL's will know upon hostory what that nodes consumed the previous time not exactly... there are some things you cannot predict... like how long it will take that algorithm to converge according to the caracteristics of the data. ok, such nodes will exsist. Those nodes will get the label 'unpredictable'. :) Let's say my neural network will take twice longer to converge if I train it with speech than if I train it with music... how will the BL know what files I'm senting to it... also, even the idea of how long it will take to converge is very intuitive. The problem is that these "unpredictable" nodes will often be the most resource consuming... in my case at least. ok, but nodes will consume resources in more-or-less the same order of magnitude maybe this is something that needs experimenting.. To me the design of having 1 central boss BL for each network will be limiting. Also, even with many Piper's running, I don't see the problem of centralizing decisions... Each MBL tells the splitter how much resources it has left, the splitter receives all that info and takes the decision. hmm... shall we continue this discussion tomorrow, it's 23:50 here. I'm a bit tired. maybe I see things much clearer tomorrow :) I'll give your point of view some brainpower tomorrow... let you know more next time, ok? brad: bye, you're around tomorrow too? Sure, yeah, I can be really quiet tommorrow as well :-) From chapmanb at arches.uga.edu Tue Sep 5 18:38:13 2000 From: chapmanb at arches.uga.edu (Brad Chapman) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] IRC log about general DL/BL/PL design Message-ID: <200009052238.SAA73240@archa12.cc.uga.edu> Hey all; The IRC chat today didn't happen as plan due to Jarl's sickness :-<. So there wasn't any time to do any arguing about design. Instead, I asked a lot of questions of Jean-Marc about the design to get myself more in touch with what he was thinking. The biggest change since yesterday is the idea of a Grandma-BL. This is a since high level BL process which communicates with the multiple mother-BLs. Jean-Marc and I talked about possibly implementing this in python in the DL and thus changing the interface that the DL uses to communicate with the BL. Anyways, the log is below for your perusal, if interested. Comments are, as always, quite welcome. Brad [Snip...talking about Jarl being sick...] I'm coding the C++ CORBA stuff for the DL/BL communication. Fun with C++ :-) BTW, any idea what's the best way to discuss about this splitting thing with everybody? Well, here is an idea. Why don't you write something about it on the piper list... Well, if we have a grandma-BL, there'll be no DL/BL Corba communication. The grandma-BL will be linked against the DL and the CORBA communication would be between the grandma-BL and the mother-BL's then start a page on the recently set up Piper Wiki (http://www.bioinformatics.org/piperwiki/)... I can do that. then we could discuss it there and people could go through it. I could set up a page for it. Then we'll have some kind of ready-made documentation on it -- or at least something to start with... grandma-BL <- So you think the GBL will be in python? I'm confused... Can't the GBL be done just like libflowui: a C/C++ dynamic library, that the python DL could "use"/talk to? I guess so, although I really prefer CORBA over simply wrapping things, since this helps keep some language independence. Additionally, I'm not sure how I would know if the GBL dies. With CORBA it shouldn't be too difficult to know it died and restart, but I don't know how this will work if I just wrap it. Well, if it's linked against the DL, then when it dies, the DL dies too! Everything would be exactly the same way you use libflowui. Plus, I don't see why use CORBA when you are on the same machine... ...If you prefer, then you could write the GBL in python and put it in the DL (the GBL would become the DL) dying <- Well, that is really bad, right? IF the DL dies then the UI will just be left hanging. I'd like the DL never to die... There's no way you could be sure the DL won't die... except if you don't run a DL at all :-) CORBA <- Well, I like CORBA a lot more than wrapping directly. I don't think the slowdown over direct wrapping is that big of a deal, but CORBA provides a nice interface to work off of, and would also make it possible to rewrite the DL (if someone wanted to) in any language (ie. java, if someone digs that). die <- Well, of course :-). But I don't want to the DL to die just because the BL dies during a run! With a CORBA interface we should be able to detect the death, and restart the BL as a separate process. Or what would you think about saying that the GBL and DL is one thing? All written in python. Then all the MBL would communicate with the DL. GBL in python <- Well, I don't know how much of the GBL functionality Jarl already has in GMS. But I wouldn't mind writing it in python at all. This is all provided that what Jarl has right now is mostly a bunch of MBLs. Do you have any kind of vision for what the GBL/MBL interface would look like? Sounds fine... but I doubt Jarl wrote the GBL, since we just "decided" there would be a GBL. The functionnalities are for viewers/probes and (maybe) splitting, none of which Jarl started coding... Not at all. I have never touched (or even approached) CORBA... I have an idea about the BBL/PL interface, but that's all! The GBL in python sounds great by me (one more reason why the dl is important :-). I have a slight idea of what the communications between the PL and the DL should look like with no idea how it will get through the BL. Well, the BBL to PL interface is definately a good start :-) ...and believe me, it won't be CORBA! I'd like to kind of see if we could at least hash out the rough ideas of how the communication will look. I'm kind of at a loss about how to proceed next with this, although I'm ready to code... no CORBA <- :-). No problems. I like CORBA a lot, but that doesn't mean everyone has to... ...not that I don't like CORBA (I don't know it enough to like it or not). I just see the BBL linked to the PL. Yeah, I agree, this is also I have been thinking about it thus far. I think the communication will be message/command based. The DL will be able to send a PL node commands like "Next", "Continue", "Stop", .. But I have also been thinking about the BL/DL communication (or GBL to BBL) as CORBA. A PL node will the the DL: "plot this data", ... The message based system makes a lot of sense with what we were talking about in terms of that iostream stuff and having the error logging system. Well, the GBL will not talk directly to the BBL's, there's MBL watching ;-) I don't have a very concrete idea of how the BBL/PL stuff will look, but what you say makes sense. MBL <- :-) Whoops sorry, I meant GBL to MBL. Still getting the new acronyms straight. the BBL will just call getOutput() on its subnetwork. If at some point a PL node needs to do a getOutput() on a remove machine, it will ask the BBL for that data. How the BBL gets it is not the PL's problem. In fact, it will end up as another BBL calling getOutput on its subnetwork. ...oops... I can't help on that! Okay, but how will the PL callback to the BBL to let it know it needs info from a remote machine? let's say we have A -> (B->C) -> D, where B->C is on a certain machine... Okay, I recognize this example :-) When breaking the network, B's input will be connected to a special communication node. This node will call a BBL function to get this data. the BBL will manage to get the data from A by talking (indirectly) to another BBL and will return it to the special node, which will return it to node B. Okay, so everything happens through the special communication node. Is your idea for the design of this similar to the interface we were kind of talking about with the iostream overriding class? no, this node will just cal a function. The only reason for the iostream class is that I would like to be able to make that stream equal to cerr (stderr) if I like without changing the code in the nodes. Okay, I'm with you. What do you imagine the function calls the node will make looking like? Would this be similar to the interface type that I sort of set up in the IDL file I sent a while back, or something completely different? It would be sonething like: theBBL->getMeMyInput(this, inputID, count); Okay, gotcha. BTW, what exactly is count? I saw it in all the getOutputs() but I never exactly got it. ("this" is a pointer to the current node, in case you're not familiar with this part of C++) Okay what about errors generated during processing a getOutput() in a node. How would those get returned to the BBL? this <- Yup. I'm with you on that :-) Thanks, tho! There's a count when loops are involved, it's used to select to output number. it depends on what kind of error... Okay, so count is only relevant when in a loop. error <- Well, what different kinds do you imagine? some errors are "fatal": they throw an exception which will be caught my the BBL. When there's an exception, it means that the task simply cannot be performed (eg. a parameter is missing to a certain node). Okay, that makes sense. What about non-fatal errors? Warnings through the streams (cerr << "something strange here" << endl;) will be sent to the DL (throught the BBL) and everything will continue. when not in a loop, the count asked will be zero; What about an error which might cause the user to want to choose between stopping and continuing (some kind of processing error)? The best example would be when treating video... if a node asks another getOutput(outputID, n), it means I want to have the n'th frame. Or how about your debugging/probe stuff? We will need to define a special interface for those... but I haven't found a use for that so far (though not saying it doesn't exist) debugging will be handled through display/next/continue commands Okay, but we should just stick with simple cases first and see when the other cases come up, you're right. I think this makes sense how the BBL/PL will happen. I guess we need to talk through everything with other people to see how the GBL/MBL should look... basically it'll be about commands (sometimes with data associated to it)... The address of a node will me something like MBL:BBL:ptr (the address of the MBL, the address of the BBL, the (C++) pointer to the node) Have you had a look at the dl2bl.idl interface Jarl and I wrote? nope... I know nothing about idl stuff, so I couldn't comment on it anyway. commands <- I think (maybe) I am with you on this. I basically understand the concept, although the implementation is foggy in my head. idl <- Well, it is very similar to C++ syntax, so it should be hard to follow. I just thought maybe this might give you ideas about the interfaces (ie. classes) we were thinking about for communicating between the DL and PL. This was written a while back, but I think parts may still be relevant to where we are at now. Ooops, that's 'shouldn't be too hard to follow' :-) Don't worry, the implementation is foggy for me too! I think the BL shouldn't need to know about the messages... The BL will just transmit "data" implementation -> Well, as we go, can we sketch it out in C++ header file syntax or pseudocode or something? This would help me follow a lot better, since I have some trouble thinking really abstractly :-). We could use the wiki for this, again. messages <- I'm a bit confused about these messages all of a sudden. Are these messages utilized by the PL? Well, there's no "detail" right now, the details will depend on the type of node (viewer/probe/...) When the DL wants to say "Next" to a PL probe, it sends the command through the BL, tough the BL never knows what "Next" means. Okay, so these are messages for DL to PL communication (and vice versa). I'm with you. Will these messages be defined in some kind of XML syntax, in your mind? Probably not... It will probably look a bit like a pseudo-packet... There will ba an address (MBL:BBL:ptr), a command and some data. The data will be a serialized Object (deriving from my Object class) Okay, but the only think I don't understand is how the ptr (the most important part) will be transmitted about, especially between python and C++. Or maybe, the command and data will be packed in an Object, which will be the data in the pseudo-packet. serialized <- In what way? Well, a DL->PL communication will always be some kind of answer to a PL->DL communication. The talking node will tell the DL its pointer. The DL won't know what to do with it, but the BBL, which is in the same address space (because it links to the PL) will. serialized: The whole Object (sometimes containing other Objects) will be changed in a string of bytes. serialized <- This is my sticking point since I don't really understand serialization :-< Sorry, I'll have to try and bone up on it. So far, I only support ASCII serialization: serializing a vector of float looks like If I have a vector of vectors, it will look like > >> The idea is just that an Object can be sent through a network or saved to a file. Okay, so this is just C++ specific serialization right? Maybe I do understand, since python has a pickle module which provides persistance like this (ie. you can 'pickle' a data structure, and then load it up from another program instance). I just don't understand it in C++ yet, but I get the idea now! It's not a C++ feature. it's just what you have to do to transmit an object... maybe python does it for you, but it still needs to do it. Right, but there is a specific syntax that you need to spit it out in so C++ can reload it (or transmit it across a network). What I mean is that I don't yet know this syntax, but I can understand what is it doing :-). How do pointers get serialized via this mechanism? I decided on the syntax... I wrote the save and the load. It's like for a UIDocument that has contains a bunch of UINetworks that in turn contain UILinks and UINodes, which contain UINodeParameters... You manage to transform that to a bunch of bytes, which to save (the .n files) For my Objects, instead of using libXML to serialize, I wrote my own serializer, which is cleaner for what I do. Okay, understood completely. Sorry to be so confused on that! I think I sort of see where we are at, now we just need to move forward and figure out the implementation stuff and see what other people think about things... but that is probably for another chat session :-) Yep! In the mean time, I'll try to expose my ideas on splitting on the list. Makes sense. I've been logging this, so I'll send it to the list as well. Thanks for talking this over with me! Ttyl. ttyl. Have a nice night! From chapmanb at arches.uga.edu Tue Sep 5 19:00:07 2000 From: chapmanb at arches.uga.edu (Brad Chapman) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Piper Wiki is up and running! Message-ID: <200009052300.TAA146370@archa11.cc.uga.edu> Hello all; Thanks to Gary's help, the Piper wiki is up and running on bioinformatics.org. You can check it out at: http://www.bioinformatics.org/piperwiki/moin.cgi It is also linked to from the mail Piper page. I've started working on adding information and will expand on this over the next few days. Since the point of wiki is to allow collaborative documentation, everyone is highly encouraged to check it out and add info/comments etc. If you have never used wiki before, the best place to start is the EditingTips page at: http://www.bioinformatics.org/piperwiki/moin.cgi/EditingTips This will give you the idea of the type of markup you can use when editing Wiki pages. Once you feel ready to try it out, head over to the WikiSandBox (linked off of the EditingTips page) where you can play around without any fear of messing things up. Enjoy! Comments are very welcome. Brad From jean-marc.valin at hermes.usherb.ca Wed Sep 6 21:03:18 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:28 2006 Subject: [Pipet Devel] Licensing issues Message-ID: <39B6E956.344C767@hermes.usherb.ca> With the latest issues/wars (depends on how you see it!) with the KDE/Qt licensing, I think it's now time to make sure not to end up with the same mess (no matter which side you are, the result is that it has hurt KDE a lot) with Piper. This is mostly important for the PL (Overflow), because CORBA linking is not covered by the (L)GPL. Also, now's the time to change the Overflow (PL) license, because all the copyrights are owned by three people: Me, Dominic, and Brad. The three need to agree to change the license (should not be too hard!). If we wait too long and too many people contribute, it'll be too late (there are hundreds of copyright owners in the Linux kernel, changing license is now impossible). For now, the license is GPL, since at first, it was not meant to be a used as library. I think we all agreed (do we?) that the basis sould be around the GPL/LGPL. Before I continue, here's my simple explanations on each. You may link any software (be it a program or a library, OSS or not) to an LGPL library, while to may not link to a GPL library. (this is commonly known). You CANNOT link either LGPL or GPL software (library or program) to a library that is not GPL-compatible (LGPL, GPL, and BSD are GPL-compatible, while the QPL is not). This is the clause that caused so much problem with KDE and that made it illegal, since libkdeui (which is LGPL) could not legally be linked to Qt, which was not GPL-compatible. Now this is what I'd like to discuss: 1. Should we allow non GPL-compatible nodes to be used (my opinion is yes)? If we want that, we need to provide with an exception to the GPL/LGPL that explicitly allows it (like KDE needed to have so it could be linked with Qt). The unclear part of this is would we be allowed to use *at the same time* non-GPL compatible nodes AND GPL nodes that don't provide the exception. 2. (apart from clause 1) Should we release the PL as GPL or LGPL? The LGPL would be less restrictive, but would also allow any proprietary Piper-like system to use Overflow as it's processing layer. What do you think? We'd also have to choose a license for the DL and BL. Right now, the DL has to be GPL, since it links with GPL code (libflowui is currently GPL) and it would be illegal to release it as LGPL. So I hope we can get the situation straight will less pain than KDE. Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jeff at bioinformatics.org Thu Sep 7 11:27:32 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> Message-ID: <39B7B3E4.2DBB52F7@bioinformatics.org> In my opinion, we could license all of the parts to Piper under either the GPL OR LGPL, providing we clarify what is meant by "linked". GPL vs. LGPL: As Jean-Marc mentioned, the LGPL ("lesser" or "library" GPL) is meant to allow closed-source programs to "link" TO LGPL-licensed code. The GTK+ and GNOME libraries are both LGPL'd, which means a company like Applix can use them for Applixware with no legal problems. (The Qt library, however, has recently been re-licensed under the GPL, which would force such a company to use the alternative QPL and pay for it.) Is Piper a library? Well, sort of. I believe applications like BlueBox, ISYS, and OpenBSA will make use of Piper in such a way. And, ISYS is closed-source, which means the GPL might not work for them. But, the fact that the use of Piper by ISYS would take place through CORBA, is an issue. Is the use of CORBA considered "linking"? Jean-Marc says no. If not, GPL vs. LGPL is irrelevant to us. Neither the GPL or LGPL provides us with a good definition of "linking". I used to find this frustrating, since so much hinges on the definition. However, I now think this was intentional, because there is an enormous continuum of "linking" examples out there. It should thus be up to the author to define or clarify what is meant by "linking" and how it applies to their code. The LGPL would cover "one end" for us, without having to specify that "CORBA is not linking": It says that pretty much anything can link TO Piper. But, we would nonetheless be left to define what can be linked FROM Piper. I say we should put an exception in the license (either GPL or LGPL) for all programs, etc., that are "linked" as "nodes". The Linux kernel has the same exception for device drivers. Maybe I didn't answer the question that Jean-Marc proposed: GPL or LGPL? I just rephrased it :-) If we all agree that the ONLY uses of Piper would be via CORBA and nodes, then we should choose the ordinary GPL. Vote? Cheers. Jeff Jean-Marc Valin wrote: > > With the latest issues/wars (depends on how you see it!) with the KDE/Qt > licensing, I think it's now time to make sure not to end up with the same mess > (no matter which side you are, the result is that it has hurt KDE a lot) with > Piper. > > This is mostly important for the PL (Overflow), because CORBA linking is not > covered by the (L)GPL. Also, now's the time to change the Overflow (PL) license, > because all the copyrights are owned by three people: Me, Dominic, and Brad. The > three need to agree to change the license (should not be too hard!). If we wait > too long and too many people contribute, it'll be too late (there are hundreds > of copyright owners in the Linux kernel, changing license is now impossible). > For now, the license is GPL, since at first, it was not meant to be a used as > library. > > I think we all agreed (do we?) that the basis sould be around the GPL/LGPL. > Before I continue, here's my simple explanations on each. > > You may link any software (be it a program or a library, OSS or not) to an LGPL > library, while to may not link to a GPL library. (this is commonly known). > > You CANNOT link either LGPL or GPL software (library or program) to a library > that is not GPL-compatible (LGPL, GPL, and BSD are GPL-compatible, while the QPL > is not). This is the clause that caused so much problem with KDE and that made > it illegal, since libkdeui (which is LGPL) could not legally be linked to Qt, > which was not GPL-compatible. > > Now this is what I'd like to discuss: > > 1. Should we allow non GPL-compatible nodes to be used (my opinion is yes)? If > we want that, we need to provide with an exception to the GPL/LGPL that > explicitly allows it (like KDE needed to have so it could be linked with Qt). > The unclear part of this is would we be allowed to use *at the same time* > non-GPL compatible nodes AND GPL nodes that don't provide the exception. > > 2. (apart from clause 1) Should we release the PL as GPL or LGPL? The LGPL would > be less restrictive, but would also allow any proprietary Piper-like system to > use Overflow as it's processing layer. What do you think? > > We'd also have to choose a license for the DL and BL. Right now, the DL has to > be GPL, since it links with GPL code (libflowui is currently GPL) and it would > be illegal to release it as LGPL. > > So I hope we can get the situation straight will less pain than KDE. > > Jean-Marc -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jean-marc.valin at hermes.usherb.ca Thu Sep 7 11:51:38 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> <39B7B3E4.2DBB52F7@bioinformatics.org> Message-ID: <39B7B98A.E57131C2@hermes.usherb.ca> > Is Piper a library? Well, sort of. I believe applications like BlueBox, > ISYS, and OpenBSA will make use of Piper in such a way. And, ISYS is > closed-source, which means the GPL might not work for them. But, the fact > that the use of Piper by ISYS would take place through CORBA, is an issue. Is > the use of CORBA considered "linking"? Jean-Marc says no. If not, GPL vs. > LGPL is irrelevant to us. If I understood what Richard Stallman had to say on the subject, linking to GPL code through CORBA (from closed-source) is legal, because it's a hole in the GPL. CORBA didn't existed (or wasn't widely used) when GPL version 2 was written. Since future versions of the GPL (remember we can link with any version newer that 2 with the current license) may patch the CORBA hole, I say: "if you want to be allowed to link with CORBA, release LGPL". > Neither the GPL or LGPL provides us with a good definition of "linking". I > used to find this frustrating, since so much hinges on the definition. > However, I now think this was intentional, because there is an enormous > continuum of "linking" examples out there. It should thus be up to the author > to define or clarify what is meant by "linking" and how it applies to their > code. Right now, linking means "static linking" and even the issue about dynamic linking isn't clean (let's assume that dynamic linking is also covered). > > The LGPL would cover "one end" for us, without having to specify that "CORBA > is not linking": It says that pretty much anything can link TO Piper. But, we > would nonetheless be left to define what can be linked FROM Piper. I say we > should put an exception in the license (either GPL or LGPL) for all programs, > etc., that are "linked" as "nodes". The Linux kernel has the same exception > for device drivers. Calling a program from inside a node is not linking. Only static/dynamic linking of code is. But the question still raises much more complex issues. For instance, can an LGPL library link to (not "be linked to") a GPL library. Let's say you have library Gtk+ was GPL instead of LGPL. Could libgnumoui (that links/uses Gtk+) be LGPL. One thing is sure is that it would be illegal to write a closed-source program that links with libgnomeui, since it would violate the GPL on Gtk+, which you need to link. Remember that this is fiction, since Gtk+ is released under the LGPL. So the real issue is: if the PL is LGPL, can it link to GPL-only libraries? Just for those who aren't familiar with it... PL nodes are classes that are part of libraries (I call them toolboxes) that are dynamically linked at startup. Even more complex is the fact that it could happen that GPL software AND closed-source software be linked from the PL is you run uses both proprietary nodes and GPL nodes. I think it would be illegal (the guy who wrote the GPL code didn't want closed-source software to be linked to it). I think we should try to wrap all our questions/doubts/fears in an e-mail and send that to either Richard Stallman or Bruce Perens (or both). What do you think? Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jeff at bioinformatics.org Thu Sep 7 11:57:25 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> Message-ID: <39B7BAE5.CD53BF1B@bioinformatics.org> Jean-Marc Valin wrote: > > 1. Should we allow non GPL-compatible nodes to be used (my opinion is yes)? If > we want that, we need to provide with an exception to the GPL/LGPL that > explicitly allows it (like KDE needed to have so it could be linked with Qt). > The unclear part of this is would we be allowed to use *at the same time* > non-GPL compatible nodes AND GPL nodes that don't provide the exception. Hmmm. I don't believe each node would need to provide the exception to the "linking restriction". It should be just Piper that provides it. However, Piper does allow "one node to be linked to another", so what are the licensing implications of that? This is a good question. I recall Richard Stallman writing at one time about the implications of Free Software being used by Application Service Providers (ASP's). The question someone posed was, "Do ASP's need to provide the source code for the Free Software a user executes via their service?" Stallman's response was that the user is not effectively "linking" to software, but is instead using it as a service, so the source code does not have to be available. The same is true for Web servers. Think about it: If you wrote a Web server and licensed it under the GPL, must every browser that accesses the server be GPL'd? Would you be required to provide the source code to every surfer who visits your site? No, of course not. I think we can make the same argument for nodes connecting to Piper and to each other: They are only providing a service to each other. They are not being linked to form a single code base. In any case, I believe the spirit of the "linking clause" in the L/GPL is that, by "linking", you are creating one code base that would result in a non-Free component restricting the use of a Free component. Piper is usable without any particular node, so I think we are in jive with that spirit. We should perhaps put some of this reasoning in our license modification. > 2. (apart from clause 1) Should we release the PL as GPL or LGPL? The LGPL would > be less restrictive, but would also allow any proprietary Piper-like system to > use Overflow as it's processing layer. What do you think? I don't know. We'll still be include-ing Overflow as a library from the BL, right? I suppose that's up to you guys, if you think Overflow might be used outside of Piper. A single license might be neater though. Cheers. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jeff at bioinformatics.org Thu Sep 7 12:12:32 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> <39B7B3E4.2DBB52F7@bioinformatics.org> <39B7B98A.E57131C2@hermes.usherb.ca> Message-ID: <39B7BE70.8B7CBEF1@bioinformatics.org> Jean-Marc Valin wrote: > > If I understood what Richard Stallman had to say on the subject, linking to GPL > code through CORBA (from closed-source) is legal, because it's a hole in the > GPL. CORBA didn't existed (or wasn't widely used) when GPL version 2 was > written. Since future versions of the GPL (remember we can link with any version > newer that 2 with the current license) may patch the CORBA hole, I say: "if you > want to be allowed to link with CORBA, release LGPL". Good point. > Calling a program from inside a node is not linking. Only static/dynamic linking > of code is. But the question still raises much more complex issues. For > instance, can an LGPL library link to (not "be linked to") a GPL library. Let's > say you have library Gtk+ was GPL instead of LGPL. Could libgnumoui (that > links/uses Gtk+) be LGPL. One thing is sure is that it would be illegal to write > a closed-source program that links with libgnomeui, since it would violate the > GPL on Gtk+, which you need to link. Remember that this is fiction, since Gtk+ > is released under the LGPL. I recognized that problem when first reading the GPL and LGPL. Let me phrase it another way to see if we're talking about the same thing: The LGPL specifically allows something (linking) that is not allowed by the GPL. Would merging LGPL + GPL code bases then violate the GPL, or at least be very confusing? I think the intention of the LGPL is to be a "lesser" license, and to thus be superseded by the GPL when the two are combined. A clause in the new version of the LGPL actually permits an "upgrading" to the GPL (I think by anyone). So, maybe when the two are combined, the LGPL automatically defers to the GPL? > So the real issue is: if the PL is LGPL, can it link to GPL-only libraries? Just > for those who aren't familiar with it... PL nodes are classes that are part of > libraries (I call them toolboxes) that are dynamically linked at startup. Even > more complex is the fact that it could happen that GPL software AND > closed-source software be linked from the PL is you run uses both proprietary > nodes and GPL nodes. I think it would be illegal (the guy who wrote the GPL code > didn't want closed-source software to be linked to it). Oh, I see what you're talking about: Overflow nodes. Would you say that they are statically/dynamically linked at compile time? > I think we should try to wrap all our questions/doubts/fears in an e-mail and > send that to either Richard Stallman or Bruce Perens (or both). What do you > think? Heh. Stallman is actually really nice about replying to e-mails. He replied to more than a couple of mine. I don't know about Perens. Eric Raymond never replied to my e-mails, even when Stallman CC'd them to him. Cheers. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jarl at casema.net Fri Sep 8 01:56:57 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> <39B7B3E4.2DBB52F7@bioinformatics.org> <39B7B98A.E57131C2@hermes.usherb.ca> Message-ID: <39B87FA9.8810BC4B@casema.net> > If I understood what Richard Stallman had to say on the subject, linking to GPL > code through CORBA (from closed-source) is legal, because it's a hole in the > GPL. CORBA didn't existed (or wasn't widely used) when GPL version 2 was > written. Since future versions of the GPL (remember we can link with any version > newer that 2 with the current license) may patch the CORBA hole, I say: "if you > want to be allowed to link with CORBA, release LGPL". The only place we've this Corba 'linking' problem is with 3th party UI's, isn't it? If we want people to be 100% free to build UI's we should go for LGPL. > > Right now, linking means "static linking" and even the issue about dynamic > linking isn't clean (let's assume that dynamic linking is also covered). Linking seems to be a though problem to us I think. Imaging this: Piper UI -> Piper DL -> '3th party BL' -> PL What if this 3th party BL registers to our central coordination database, or what if it screws to authentication meganism? It can be done very easily if one were to recode some core parts of the BL. Like has happend to icq. Clones line gnomeicu 'parasite' upon mirabilis. gnapser this to Napster. etc. I whould therefore see we licence our communication formats (like the XML format, corba API) very strict! Something that would allow people to build UI's, link TO Piper (not link Piper INTO some other app), and 'hook' apps into the PL. I see much chaos wonce parts of piper are to be 'emulated' by some 3th party. bye, jarl From jean-marc.valin at hermes.usherb.ca Thu Sep 7 17:28:04 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> <39B7B3E4.2DBB52F7@bioinformatics.org> <39B7B98A.E57131C2@hermes.usherb.ca> <39B87FA9.8810BC4B@casema.net> Message-ID: <39B80864.E38AE131@hermes.usherb.ca> > The only place we've this Corba 'linking' problem is with 3th party UI's, isn't it? There might be some other cases, but I think 3rd part UI's would be the most important. > > If we want people to be 100% free to build UI's we should go for LGPL. This means the DL should be LGPL, but the BL's than run on each machine could be GPL. ...it all depends on how you see the linking... with remote comunications, this becomes very blured. For instance, you can't ask that the user of a GPL'd ftp client only connects to GPL'd ftp servers... of that mozilla (which is GPL) only goes to sites that run apache. The same way, I don't think we can forbid non-free BL's that can be connected to the rest of the system (even if everything else is GPL'ed). > Piper UI -> Piper DL -> '3th party BL' -> PL > > What if this 3th party BL registers to our central coordination database, or what > if it screws > to authentication meganism? It can be done very easily if one were to recode some > core > parts of the BL. Like has happend to icq. Clones line gnomeicu 'parasite' upon > mirabilis. > gnapser this to Napster. etc. > > I whould therefore see we licence our communication formats (like the XML format, > corba API) very strict! > Something that would allow people to build UI's, link TO Piper (not link Piper INTO > some other app), and > 'hook' apps into the PL. I see much chaos wonce parts of piper are to be 'emulated' > by some 3th party. We cannot (and shouldn't) prevent 3rd party BL's. The only way to do that is through proprietary (patented) protocols and we don't want to do that. As for the authentication mecanism, it has to work for any (even hostile BL's). Napster had problems because it's assumed all its clients were the official clients. This kind of security is flawed, even more in OSS. If a 3rd party BL can screw up the authentication, then nothing prevents a cracker to modify your BL source to screw it up. This means that your BL is not secure. This is why we sould not try to prevent 3rd-party BL's (that's the whole point of open-source: someone can modify your work). However, if the BL links with the PL, then depending on the PL license, a closed-source BL may or may not be legal... but then they could provide their own PL and there'd be nothing we can do. Before we start going into license (in)compatibilities, let's start by deciding what kind of things we want to prevent. We know we don't want people to take our code and make it proprietary. That leaves us with GPL and LGPL. Now is there any kind of "linking" we want to prevent? If not, then we can almost LGPL everything. If not, we need to see what needs to be GPL and what needs to be LGPL. Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jarl at casema.net Fri Sep 8 02:49:59 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> <39B7B3E4.2DBB52F7@bioinformatics.org> <39B7B98A.E57131C2@hermes.usherb.ca> <39B87FA9.8810BC4B@casema.net> <39B80864.E38AE131@hermes.usherb.ca> Message-ID: <39B88C17.BBBCC457@casema.net> > We cannot (and shouldn't) prevent 3rd party BL's. The only way to do that is > through proprietary (patented) protocols and we don't want to do that. As for > the authentication mecanism, it has to work for any (even hostile BL's). Napster > had problems because it's assumed all its clients were the official clients. > This kind of security is flawed, even more in OSS. If a 3rd party BL can screw > up the authentication, then nothing prevents a cracker to modify your BL source > to screw it up. Do you happen to know if some firm that has this problem somehow took legal action against this. Or is this not possible? (I remember this aol vrs M$ struggle about the M$ messenger and Icq. How did thius end, or is this still going on?) > > This means that your BL is not secure. This is why we sould not > try to prevent 3rd-party BL's (that's the whole point of open-source: someone > can modify your work). However, if the BL links with the PL, then depending on > the PL license, a closed-source BL may or may not be legal... but then they > could provide their own PL and there'd be nothing we can do. Ok, I see the problem. Legal stuff gives me headaches :) You're right about us fixing our own problems. bye, jarl From jeff at bioinformatics.org Thu Sep 7 18:12:32 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> <39B7B3E4.2DBB52F7@bioinformatics.org> <39B7B98A.E57131C2@hermes.usherb.ca> <39B87FA9.8810BC4B@casema.net> <39B80864.E38AE131@hermes.usherb.ca> Message-ID: <39B812D0.60115D17@bioinformatics.org> Jean-Marc Valin wrote: > > > The only place we've this Corba 'linking' problem is with 3th party UI's, isn't it? > > There might be some other cases, but I think 3rd part UI's would be the most > important. Regardless of whether we choose the LGPL or GPL, we can always make a modification that explicitly permits CORBA linking or any other kind of linking. That's something we should keep in mind. > Before we start going into license (in)compatibilities, let's start by deciding > what kind of things we want to prevent. We know we don't want people to take our > code and make it proprietary. That leaves us with GPL and LGPL. Now is there any > kind of "linking" we want to prevent? If not, then we can almost LGPL > everything. If not, we need to see what needs to be GPL and what needs to be > LGPL. Jarl has a good point in that we may want to prevent one of the layers from being substituted with a proprietary program. This may be more of an advocacy issue than a security issue, since, as Jean-Marc stated, an Open Source layer can be just as malicious. On the other hand, there may be some "control" issues here. Let's pretend that Micro$oft (perhaps we are having some delusions of grandure) wants to "Embrace, Extend, and Extinguish" Piper. It could do so by creating its own BL (or another layer) that uses a SLIGHTLY different protocol that only works with their UI. They've done this before: Look at how many Web pages work only with IE. But, if we are to make everything GPL, and say that CORBA linking from a non GPL'd layer is illegal for DL <-> BL <-> PL linking, then it has to be illegal for UIL <-> DL linking too. It's a tough choice. We (at least Jarl and I) want the former to be illegal but the latter to not be. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jean-marc.valin at hermes.usherb.ca Thu Sep 7 20:47:38 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> <39B7B3E4.2DBB52F7@bioinformatics.org> <39B7B98A.E57131C2@hermes.usherb.ca> <39B87FA9.8810BC4B@casema.net> <39B80864.E38AE131@hermes.usherb.ca> <39B812D0.60115D17@bioinformatics.org> Message-ID: <39B8372A.B95ACFE7@hermes.usherb.ca> > Regardless of whether we choose the LGPL or GPL, we can always make a > modification that explicitly permits CORBA linking or any other kind of > linking. That's something we should keep in mind. In their current form, the GPL and LGPL allow linking with CORBA. > > > Before we start going into license (in)compatibilities, let's start by deciding > > what kind of things we want to prevent. We know we don't want people to take our > > code and make it proprietary. That leaves us with GPL and LGPL. Now is there any > > kind of "linking" we want to prevent? If not, then we can almost LGPL > > everything. If not, we need to see what needs to be GPL and what needs to be > > LGPL. > > Jarl has a good point in that we may want to prevent one of the layers from > being substituted with a proprietary program. This may be more of an advocacy > issue than a security issue, since, as Jean-Marc stated, an Open Source layer > can be just as malicious. Well, if you have a (static or dynamic) linking, then you can prevent that. Otherwise, you can't. For instance, You cannot release under the GPL an ftp client for which you forbid to connect to a closed-source ftp server. I would go as far as saying that such a restriction would make your license non GPL-compatible. Actually, locking such 3rd parties would be playing the very same game as all the big bad software vendors. ...It would be like having the gcc license saying you cannot compile closed-source programs with it! > On the other hand, there may be some "control" issues here. Let's pretend > that Micro$oft (perhaps we are having some delusions of grandure) wants to > "Embrace, Extend, and Extinguish" Piper. It could do so by creating its own > BL (or another layer) that uses a SLIGHTLY different protocol that only works > with their UI. They've done this before: Look at how many Web pages work only > with IE. Actually, locking out 3rd parties implementations would be playing the very same game as all the big bad software vendors. The only thing we could try doing is to patent the protocol and allow free use as long as you don't make a protocol modification proprietary. It would be like a GPL-protocol, but I'm really not sure we can (and want to) do that. > But, if we are to make everything GPL, and say that CORBA linking from a non > GPL'd layer is illegal for DL <-> BL <-> PL linking, then it has to be illegal > for UIL <-> DL linking too. It's a tough choice. We (at least Jarl and I) > want the former to be illegal but the latter to not be. In theory, you could allow CORBA linking for some parts, and not for others... However, as I stated above, the GPL and LGPL *allow* CORBA linking from and to closed source programs and I think adding a restriction would make Piper GPL-incompatible and we do not want to do that. You can allow more that stated in the (L)GPL, but you cannot add restriction. The "allow CORBA exception" was only to clarify the matter, but right now it is legal anyway. Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jeff at bioinformatics.org Fri Sep 8 07:20:46 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] Licensing issues References: <39B6E956.344C767@hermes.usherb.ca> <39B7B3E4.2DBB52F7@bioinformatics.org> <39B7B98A.E57131C2@hermes.usherb.ca> <39B87FA9.8810BC4B@casema.net> <39B80864.E38AE131@hermes.usherb.ca> <39B812D0.60115D17@bioinformatics.org> <39B8372A.B95ACFE7@hermes.usherb.ca> Message-ID: <39B8CB8E.98624A1A@bioinformatics.org> Jean-Marc Valin wrote: > > You can allow more that stated in the (L)GPL, but you cannot add restriction. True, I did not consider that. However, we must keep in mind that at some point CORBA linking may become illegal for the GPL. If we choose the GPL, we can specifically permit CORBA linking for certain cases. Other cases could then be left illegal for non-GPL programs (3rd party BL, for example). But, do we want to wait? Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jarl at casema.net Sat Sep 9 07:16:42 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] BL->PL design Message-ID: <39BA1C1A.D3AD6394@casema.net> Hi All, here the irc log between Jean-Marc and me. It's about the BL->PL design. There's might be info missing because there's a lot discussed via email. JM offered to write a document about the design us two are happy about. ok, I think even on the bandwidth cq network delay issue we agree. My point is that the user will know where the network bottlenecks will be, the BL won't somewhere is a turning point though, when 200% ram is requested for say, the schedular tends to de-lacalisation What do you mean? localisation should be the 1th thing the schedular should distribute.. de-localisation = distribute node to remote machine but I would like to pute more logic into the schedular then just 'do what the xml says' Well, there are some issues that don't distribute well... if you consider a network which will be network-bound and not CPU(or RAM)-bound jep.. maybe we can weight total network execution time? Finding a split that minimizes the network links is a bit like solving the traveling salesman problem. traveling prob <- that's why I promote this non-localised methode It's already an NP problem... if you try to distribute it, it just won't work. but seems we're agreed on most... maybe only details now :) or do you see more issues not ok? Well, I think deciding on how the splitting will be done is an important issue. :) but I think you have want you need, or what you're missing? or see an obstacle? what do you mean? the monitoring, the naming, the corba stuff hmm.. what more had been discussed? the scheduling... it's being optional you still see a need for a 'central BL' meganism? By scheduling, you mean decide which node runs where, right (I called that splitting)? jep, splitting = scheduling Yes, I still think we really need a central scheduler. darn :) why? It will already be hard enough to have a good scheduler... I don't see have we can make a good, distributed scheduler ...sorry "HOW we can make..." hmm.. the optimization has to be done globally... You cannot just the BL's take what they want. I think it can be done oops... "just LET the BL's take..." not if you take into account the network. even if we assume all outputs have the same size This is equivalent to "solve the traveling salesman problem with distributed machines that don't talk to each other" kinda hard to explain why I think very simple rules will do the trick I'll link nodes (their execution binary or xml text) to execution times.. when the local machine is free it will go there.. such rules very simple rules that have local impact no 'all descibing' mathamatics Can you explain me how? forget my last line, it was supposed to go somewhere else! it wont be perfect, but it will fluctuate around the optimum wonce you introduce enough weighted values into the algorithm execution time ram load cpu load bandwidth all such issue are simply calculated with constanf factors and indicate where the node runs. local execution having the biggest multiplyer I don't understand you objection to running the scheduler locally... Or could we start with a local scheduler and when (or if) we need a distributed one, we can write it? sure but I think this is usage of piper, should be some options in the UI for defnining where and how you want to run This is how I see the optimal (though not necessarly practical, but still) scheduler: (I'm assuming that all required resources are known) You first need to have a function that, given the node districution(scheduling) returns how long it would take to run. given on history? Then you use an optimization technique (most likely genetic algorithms) to find out there the global minimum of the function is. ie what's the scheling that minimizes the total time it takes to run the "program" This can only be done if the "evaluation function" knows where each node is located. ok. so the BL that gets the xml data uploaded from the DL should do ALL the scheduling and then distribute? Yes. That's what I think. needs much cpu power.. needs uploading of recources useage back to central host? That's the only way to solve the problem. Not all problems distribute well... otherwise, all the super-computers would have ween replaced by Beowolf clusters... I think GP could do well, but not really real time. not the resource usage, but the resources available on each system (or maybe we're not talking about the same thing) Well, the genitic algorithm part could distribute some crunching to the other machines if it's that complicated... but the problem itself is resolved locally. Also, I think that if the system is too complex for a local scheduler to run fast, it will totally confuse a distributed scheduler anyway. I think the question is changed a bit: what will we do 1st. I must admit this GP distributor is quite nice, but I hope to see these rule-set based version running too I say we start with two modes for now: 1) the user does the scheduling and 2) local scheduling... later if we can try a distributed one if you really like. rule based scheduling... how? confuse a distributed scheduler? all confusion is taken care of in distribution of next network, so it will 'stabalize' around some situation Confuse in the sense that the "solution" found by a distributed scheduler will be so non-optimal... ...what you might as well distributed at random hehe speak again once it's running :) ok, but at first we wont have an GP schedular, so maybe we should 1st work on the 100% defnined version 100% defnined version? and when all the part commnunicate we'll work on a more sofistaced one I said the GP scheduler would be the optimal scheduler, but of course we shouldn't start with that. 100% definied <- all node have a fixed location\machine OK, 100%-defined first... I think the best system would be an interactive scheduler. One that talks to the UI and helps the user distributing. and also ok with this (very simple) PL main Corba thinggy? talks <- yer :) The UI could say: "I think you're putting too much stuff on machine XYZ". would be very very sweet is this voice regoc ready for such? This would give you the real optimal scheduler. (remember that the GP thing was optimal if the system knew everything about required resuirces... the user already has a better idea of what nodes do) ic ...talk in the weak sense... the interactive scheduler communicates with the UI Do we agree on most of the system? think so for a while at least :-) it's just about what goes 1st I think about the PL, did you agree if we called it Baby-BL and that it would link to the PL? (in my e-mail) the part I called PL main Corba thinggy? Something like that... ok, as long as you agree with using corba between BL and PL\BabyBL Sure... the way the BL communicates with it's other parts is up to you. I don't even know how to write an ftp client, so I'm not going to tell you what to do for this part! L) but I'll need you input for this baby BL. ...You may not like the naming... but the local scheduler is actually what I was calling grandma-BL (after all, it's responsible for the brokering) "I'll need you input for this baby BL" what do you mean? pointing me at the enrty point of the Overflow library how it can be wrapped best Sure, I'll help. but it's quite easy to understand. ok Do you have the Overflow (PL) code downloaded? it's something for later.. i'm currently buzy working on the C++ port of BL downloaded <- yes It's as simple as calling: UIDocument *doc = new UIDocument(argv[1]); doc->load(); doc->run(param); This is the actual code in batchflow.cc that starts the processing... this run, is this only called inside the last node? ir is a UIDocument a complete network? run calls getOutput() on the last node... and everything starts... a UIDocument is the equivalent of an .m file in matlab. However, when distributing, the big document will be split in several smaller UIDocument. Each one will be sent to a Baby-BL yes does each UIDocument need some special nodes? Not in overflow... but for Piper, we will need special nodes for all the inputs of a (splitted) UIDocument. This special node will call a Baby-BL function to get the result from another part of the program. as in my A->B->C example, when we decide that B->C goes on the different machine than A, we need to "rewrite" the part as S->B->C, where S is a special node that will ask the BL to fetch the result from A. ic routing the lose ends true networks ?? lose ends <- end of node chain on one machine, a splitted network that's located over multiple machines and the up- and downloading ofcourse hmm.. no, no uploading only result uplodaing darn sorry.. yes, the only "special node" required will be for result uploading no downloading extra nodes , but nodes that transport the results back yes ...well, I'm all mixed up with the up-down point of view... the special node is at the input of the network. o, why there? nah, never mind but anyways, the DL will be suppying this extra node also? the Baby-BL receives a getOutput CORBA "signal" (don't know how to call it). it calls getOutput() on its PL network... some computation is done and the special input node will call a (BBL) callback function, which will call getOutput() on another BL. ok get it Extra DL nodes... not really. There will be debugging/viewing nodes I don't know if we should say they are DL, GUI, or Pl nodes... uhhh I meant to say was if the DL add this extra node into the xml description? However (I don't know whether you were refering to that), there will be a communication system between the DL and the PL, so that nodes can send warnings/errors to the user. ...sorry, now I understand what you mean!... ok.. will this commenction go directly from DL to PL? conn... arf I'm not sure whether it'll be the DL job, the scheduler's, or the BL... it's 3:40 now :( I mean the DL/BL/... job to add the node to the document... I think it should be the scheduler... ok BL but I'll let you sleep now...... hehe nah good idear Do you mean you're still OK to discuss? ...or the "nah" was for something else? sure, I just make much more spelling errors and bollocks talk at there times :) "nah" was a out-lout thinking.. but not yet sleeping that is Anyway... who will add the "BL2PL" node is a detail. right I was talking to Brad about the DL-PL communication system. I'm just asking some to get the picture of what the babyBL will be like We though we could have "pseudo-packets" that could be transmitted by the BL. sure The Baby-BL will be quite simple. simple <- nice The DL address will be simple: "DL" since there's only one. A node address will be like "machine:BabyBL ID:node pointer" maybe we could use some sort of number system that's used Piper wide that defines locations of nodes\dl's\etc? the machine could be an IP of a name, the BabyBL ID is used to discriminate many BBL's on the same machine. Since the BBL lives in the same address space as the PL, node pointers are OK. right What sort of number system? something like this you're just describing.. DL->BL we introduced this URI numbering I described in my email... which e-mail? I dont really care how, as long as we use the same numbering uhh... mom..I'll copy paste #Define Piper URI id number (Univ. Relocation Identification, every Piper part can be access by this number: 342.0.0 = BL or UI instance, I'm not sure about this one. 342.168.0 = Subnet (or user space) 168 on machine 342 342.168.4281 = Node 4281 inside network 168 on machine 342. just a scretch, neven really critised Well, I think it's equivalent to what I was saying, right? BTW... I've just thought about something you might not like to hear... all this translation to C++ you've been doing is for what I call the mother-BL (the one that does the authentication), right? yes originaly because it was to link to Overflow :( ...if this part only communicates to the DL and the Baby-BL (thus the PL) though CORBA, maybe the translation to C++ wasn't necessary (or am I just completly lost)? hehe kinda sad :) Is the translation complete or are you still having trouble with CORBA? but it's much cleaner alfter all to have one Orb library used and so ...then you at least gain something. Brad helped my out with some examples I just finished the removal of the old corba code, and will do the new ones starting tomorrow ...so you're staying with C++? If so, have you started using the STL. If it's not that hard, it would be nice to slowly replace glib by it. C++ COrba is much more simple as C bindings luckily if I had to go the other way around it woud be much worst nah, I havent started using any C++ feature.. I just worked hard on letting the C++ compiler to shut up but I hope to learn while coding I wouldn't like to have to translate Overflow to C. I use (and need to) just about all the C++ features that C doesn't have (templaces, virtual functions, double inheritence, ...)! maybe a lot of old lines can be replaced with much better functions but for now I know C and related libraries... wont speed up things if I was to do everything in advanced C++ yet Sure... makes sense... when you know more and have some time it might be a good idea jer, slowly we learn When you're done with the C++ CORBA thing, we could start writing the BabyBL together. ok, maybe you can code the logic \ skeleton, and I can code the corba portals? Sure, as I said, the way I see it so far, the non-CORBA part will probably be around one hundred lines of code. bye, jarl From chapmanb at arches.uga.edu Sat Sep 9 12:47:06 2000 From: chapmanb at arches.uga.edu (Brad Chapman) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] BL->PL design In-Reply-To: <39BA1C1A.D3AD6394@casema.net> Message-ID: <200009091647.MAA89196@archa13.cc.uga.edu> Hey thanks for posting this Jarl! A couple of quick comments below: > JM offered to write a document about the design us > two are happy about. I would *very* much like to see this. Right now I am really lost as to where Piper is going next and what stuff we should be coding on. > I meant to say was if the DL add this extra node into the xml > description? I'm not exactly sure if this is what you guys were talking about here, but right now the XML description that the DL outputs should be immediately runnable by the PL. Recently I wrapped the UIDocument::run() function and ran a couple of XML descriptions produced by the DL through this and they ran without problems. This is definately not perfect, but should work for most cases. > A node address will be like "machine:BabyBL ID:node pointer" > maybe we could use some sort of number system that's used Piper > wide that defines locations of nodes\dl's\etc? I'd be really interested to hear what we are going to do for this point. Right now we've got a numbering system for the dl2bl communication (which Jarl talked about in the IRC) but I don't know if this is completely compatible with what Jean-Marc wants needs. In my mind it would be very useful to create a standard representation of a node that we could use throughout Piper and that could be serialized and unserialized by all of the different parts. Since my last IRC conversation with Jean-Marc I've been thinking a lot about this and how important it is to have a way to pass these nodes around between the different parts. I don't know if I really know the best way to do this, but I thought I would throw it out to see if anyone has any suggestions/thoughts on it.... Thanks again for posting this. I'm glad that Jarl and Jean-Marc are coming together on a design they both like :-). Brad From jeff at bioinformatics.org Sat Sep 9 13:02:16 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] BL->PL design References: <200009091647.MAA89196@archa13.cc.uga.edu> Message-ID: <39BA6D18.EC2EC834@bioinformatics.org> Brad Chapman wrote: > > > JM offered to write a document about the design us > > two are happy about. > > I would *very* much like to see this. Right now I am really lost as to > where Piper is going next and what stuff we should be coding on. Beyond the many goals of Piper that we have already discussed (on this list and in documents), at the lowest levels of detail, we'll have to do things democratically, with a lot of discussions between us. I think it is beyond any one of us to dictate the how every piece of code need be written. Even Linus Torvalds doesn't hand out coding assignments to contributors :-) I'm happy to see the IRC conversations and some of the details for DL <-> BL <-> PL interaction being worked out. This is surely how we should keep proceeding. Cheers. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jean-marc.valin at hermes.usherb.ca Sat Sep 9 13:09:05 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] BL->PL design References: <200009091647.MAA89196@archa13.cc.uga.edu> Message-ID: <39BA6EB1.BE9004CE@hermes.usherb.ca> > > JM offered to write a document about the design us > > two are happy about. > > I would *very* much like to see this. Right now I am really lost as to > where Piper is going next and what stuff we should be coding on. Oops.. perhaps I got misunderstood... I don't really have time for that... at least not until I come back from Chicago (Cpeech Coding Workshop... not everybody is doing bioinformatics :-) ) on the 20th. > > > I meant to say was if the DL add this extra node into the xml > > description? > > I'm not exactly sure if this is what you guys were talking about here, > but right now the XML description that the DL outputs should be > immediately runnable by the PL. Recently I wrapped the UIDocument::run() > function and ran a couple of XML descriptions produced by the DL through > this and they ran without problems. This is definately not perfect, but > should work for most cases. What Jarl means is that when we break up a network in parts, all the inputs that are not connected anymore will have to be connected a a special node that will ask for the input to the BL. What he asked was: sould the DL add this node before passing the sub-network to the PL of is it somebody else's job. (This is not a major issue right now... and we'll figure that out later) > > > A node address will be like "machine:BabyBL ID:node pointer" > > maybe we could use some sort of number system that's used Piper > > wide that defines locations of nodes\dl's\etc? > > I'd be really interested to hear what we are going to do for this point. > Right now we've got a numbering system for the dl2bl communication (which > Jarl talked about in the IRC) but I don't know if this is completely > compatible with what Jean-Marc wants needs. In my mind it would be very > useful to create a standard representation of a node that we could use > throughout Piper and that could be serialized and unserialized by all of > the different parts. Since my last IRC conversation with Jean-Marc I've > been thinking a lot about this and how important it is to have a way to > pass these nodes around between the different parts. I don't know if I > really know the best way to do this, but I thought I would throw it out > to see if anyone has any suggestions/thoughts on it.... I think Jarl's numbering scheme is roughly the same as what I am suggesting, but all with numbers. I think we agree on the principle of a uniform address scheme. Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jeff at bioinformatics.org Sat Sep 9 13:58:48 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] where is piper going? References: <200009091647.MAA89196@archa13.cc.uga.edu> Message-ID: <39BA7A58.A0678E20@bioinformatics.org> Where is Piper going next? Who knows? Actually, we do have a pretty clear idea, but we want to keep our options open. Historically, Piper's design has always been in a state of flux, and I think that is a Good Thing. Piper started out (from the "Loci" end) as "The Lowell Package". It was supposed to be "the GNU equivalent of The Wisconsin Package (or GCG)", a commercial, mostly console-based, suite of sequence analysis tools. I don't even know if there is anyone on this list who was with me back then. And then about one and a half years ago, I discovered that Piper (under the new acronym "TULIP") had the same design goal as a project called "EMBOSS": http://www.uk.embnet.org/Software/EMBOSS/ I spent a bit of time afterward concerned that we would be duplicating the efforts of another GPL'd project. Anyone who knows me knows that that is what drives my decision making with respect to my own projects. So, I decided that we should not concentrate on tools, as EMBOSS did, but on an infrastructure for distributing tools. Shortly thereafter, I decided to not make Piper (which became "Loci" after TULIP) specifically for bioinformatics, for reasons some here have heard over and over: A general-purpose system would attract more users, and thus developers, which would be better for bioinformatics users in the long run. We would not have Jarl (GMS) and Jean-Marc (Overflow) collaborating with us had I not made that decision. And, just prior to GMS and Overflow merging with Loci, Piper was only going to provide interfaces between existing programs. Overflow introduced the whole concept of "small nodes" to us (and Loci would provide "nodes over the Internet" for Overflow). So, Piper (from any end) has been through a lot of changes. Had a design be written immutably in stone, none of us would be reading this e-mail right now. FLEXIBILITY, rather than a pre-defined path, has been, and always will be, most important. But, our objectives are becoming clearer. We know at this point that Piper is not a Post-It Notes program :-) We have written a number of documents on the overall objective. Check out the poster written by Brad: http://www.bioinformatics.org/bradstuff/ismb2.ps (warning: very large) Along with the e-mail I sent to Cameron Laird about Piper vs. .NET, we have 2 very recent and clearly thought out design documents. But things can still change. What should we be coding next? Actually, I should be used to these questions from Brad. He told me very early on that he's a code monkey and not a coordinator. What we should code on next is whatever anyone thinks will help us reach the most recently drafted design goals. I can't dictate the coding any closer than that, and I probably shouldn't. We've had some heated arguments at the times I told Brad what should be done with his code. Plus, I don't have the depth of knowledge that Jean-Marc and Jarl have regarding how their particular code bases can be developed. I hope everyone understands my position on the direction of Piper. I don't want anyone to get the impression at any time that I have abandoned the project or that it is in complete chaos. We're all too dedicated and smart for that. Cheers. Jeff -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jarl at casema.net Sun Sep 10 00:20:06 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] where is piper going? References: <200009091647.MAA89196@archa13.cc.uga.edu> <39BA7A58.A0678E20@bioinformatics.org> Message-ID: <39BB0BF6.8BAA2284@casema.net> > Where is Piper going next? Who knows? Actually, we do have a pretty clear > idea, but we want to keep our options open. I think there're are three objectives attracted to form what Piper will be: 1. The needs to have a tool that links application to each other 2. A network of nodes that are executed in a powerfull environment 3. A platform that offers a better way of building a system on top Think everybody can link these points to specific Piper participants :) > > What should we be coding next? Actually, I should be used to these questions > from Brad. He told me very early on that he's a code monkey and not a > coordinator. What we should code on next is whatever anyone thinks will help > us reach the most recently drafted design goals. I can't dictate the coding > any closer than that, and I probably shouldn't. We've had some heated > arguments at the times I told Brad what should be done with his code. Plus, I > don't have the depth of knowledge that Jean-Marc and Jarl have regarding how > their particular code bases can be developed. We still need to finish the 1st phase of creating Piper: getting a pilot to actually run :) bye, jarl From jarl at casema.net Sun Sep 10 00:23:06 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] BL->PL design References: <200009091647.MAA89196@archa13.cc.uga.edu> <39BA6EB1.BE9004CE@hermes.usherb.ca> Message-ID: <39BB0CAA.53E82455@casema.net> > > > > I would *very* much like to see this. Right now I am really lost as to > > where Piper is going next and what stuff we should be coding on. > > Oops.. perhaps I got misunderstood... I don't really have time for that... at > least not until I come back from Chicago (Cpeech Coding Workshop... not > everybody is doing bioinformatics :-) ) on the 20th. OK, that leaves this work to me :( I'll hope to have it done tomorrow.. > I think Jarl's numbering scheme is roughly the same as what I am suggesting, but > all with numbers. I think we agree on the principle of a uniform address scheme. Thinking same thing, JM and I (and Brad because he participated in the dl->bl scretch api) want to have the same thing: a Piper wide relocation id. I dont really care how it will be actually implemented, as long as it will be there. From jean-marc.valin at hermes.usherb.ca Sat Sep 9 15:28:55 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:29 2006 Subject: [Pipet Devel] where is piper going? References: <200009091647.MAA89196@archa13.cc.uga.edu> <39BA7A58.A0678E20@bioinformatics.org> Message-ID: <39BA8F77.BF778031@hermes.usherb.ca> I would just like to state my opinion on "where is piper going?". I think the answer will be much clearer once we have at least a basic integration of all the layers. When we have the GUI, DL, BL and PL talking together, we'll see what it can do and what would be nice to have. Until we reach that point, I don't think we should decide anything. I don't know how stated this law of development: "Make it work, make it right, make it fast". We're still in the make it work phase. Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jean-marc.valin at hermes.usherb.ca Sat Sep 9 15:47:53 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] Baby-BL implementation Message-ID: <39BA93E9.507EDEC6@hermes.usherb.ca> This is mostly for Jarl and Brad, but I'd like to keep the others informed: I've attached how I see the Baby-BL (which links to the PL) being implemented. This is short and I also expect the full implementation to be fairly small too. ...at least before we start putting all kinds of features. (When some comments are replaced by actual code, it could be the first working version of the Baby-BL) Comments are welcome. Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca -------------- next part -------------- #include #include int main() { //BL and PL initialization //get the network to run and the parameters from the mother-BL ParameterSet params = ...; UINetwork netToRun = ...; Network *net = netToRun->build("MAIN", p); net->initialize(); //Wait for corba events and call the corresponding methods } //This method is called remotly using CORBA ObjectRef getOutput(int output_id, int count) { return net->last_node->getOutput(output_name, count); } //This method will be called by the 'Special PL node' that will be automatically connected // to dangling inputs after the scheduling (network splitting) is done. ObjectRef fetchInput (URI node_location, int output_id, int count) { //Ask the BL to ask getOutput to the BL that owns the node specified by 'node_location' return result. } From jean-marc.valin at hermes.usherb.ca Sat Sep 9 15:53:15 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] oops... this one instead Message-ID: <39BA952B.AC723CDE@hermes.usherb.ca> -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca -------------- next part -------------- #include #include //This Baby-BL program will be called by the mother-BL that runs on each machine. //This executable will be linked against the PL libraries. int main() { //BL and PL initialization //get the network to run and the parameters from the mother-BL ParameterSet params = ...; UINetwork netToRun = ...; //Create a runnable network from the UINetwork. Network *net = netToRun->build("MAIN", p); //perform initialization net->initialize(); //Wait for corba events and call the corresponding methods } //This method is called remotly using CORBA ObjectRef getOutput(int output_id, int count) { return net->last_node->getOutput(output_name, count); } //This method will be called by the 'Special PL node' that will be automatically connected // to dangling inputs after the scheduling (network splitting) is done. ObjectRef fetchInput (URI node_location, int output_id, int count) { //Ask the BL to ask getOutput to the BL that owns the node specified by 'node_location' return result. } From jarl at casema.net Sun Sep 10 05:27:50 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] Baby-BL implementation References: <39BA93E9.507EDEC6@hermes.usherb.ca> Message-ID: <39BB5415.8F3AD17F@casema.net> > I've attached how I see the Baby-BL (which links to the PL) being implemented. > Comments are welcome. Thnx for this jm, it's a bit early for me to go too much into detail of the baby BL code. Looks good what you attached. I think the BBL is indeat goin to be a very small program. Some corba code, some small function set and we're done. From jean-marc.valin at hermes.usherb.ca Sun Sep 10 02:14:17 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] GPL/LGPL issues Message-ID: <39BB26B9.B2F08439@hermes.usherb.ca> Hi, I am a developer working on the Piper (http://bioinformatics.org/piper/) project. We are at the point where we should decide which of the Free licenses (GPL, LGPL) to use for the project. However, we find that there some unclear (to us) issues with the (L)GPL when it comes to what is defined as "linking". The first one involves a library, libdata-flow which uses plugins by opening other shared libraries using dlopen(). Is that considered linking? If libdata-flow's license is the LGPL, would it be allowed to open a GPL plugin? would it be allow to open a closed-source plugin? (or both at the same time?) Does the license of the program that links to libdata-flow change the answer to the previous questions? Our second issue has to do with CORBA. Is it allowed by the GPL to use CORBA to link a GPL'd program and a closed-source program? I hop you can help us resolve our licensing problems. Thanks, Jean-Marc Valin -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jarl at casema.net Mon Sep 11 03:09:14 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] BL PL design documentation Message-ID: <39BC851A.45A9BE44@casema.net> Hi All, As promised I've made a draft about the BL & PL design. Please supply me with anything you can think of that I missed, so I can finalize the document soon. bye, jarl -- http://sunsite.auc.dk/gms -------------- next part -------------- A non-text attachment was scrubbed... Name: BL_PL.html.tgz Type: application/octet-stream Size: 9817 bytes Desc: not available Url : http://bioinformatics.org/pipermail/pipet-devel/attachments/20000910/af9bcf42/BL_PL.html.obj From jeff at bioinformatics.org Sun Sep 10 19:19:54 2000 From: jeff at bioinformatics.org (J.W. Bizzaro) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] BL PL design documentation References: <39BC851A.45A9BE44@casema.net> Message-ID: <39BC171A.FE6CE4D2@bioinformatics.org> Question: What about the "Grandma BL"? Also, do you really want to call these components "Grandma, Mother, and Baby"? It's perhaps a little corny :-) I would prefer "Primary, Secondary, and Tertiary". Then you can abbreviate them "PBL, SBL, and TBL" or "BL1, BL2, and BL3" or whatever. Jeff jarl van katwijk wrote: > > Hi All, > > As promised I've made a draft about the BL & PL design. Please > > supply me with anything you can think of that I missed, so I can > > finalize the document soon. -- J.W. Bizzaro jeff@bioinformatics.org Director, Bioinformatics.org: The Open Lab http://bioinformatics.org/~jeff "Injustice anywhere is a threat to justice everywhere." -- Martin Luther King, Jr. -- From jean-marc.valin at hermes.usherb.ca Mon Sep 11 00:46:29 2000 From: jean-marc.valin at hermes.usherb.ca (Jean-Marc Valin) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] BL PL design documentation References: <39BC851A.45A9BE44@casema.net> <39BC171A.FE6CE4D2@bioinformatics.org> Message-ID: <39BC63A5.FEC42336@hermes.usherb.ca> > Question: What about the "Grandma BL"? > > Also, do you really want to call these components "Grandma, Mother, and > Baby"? It's perhaps a little corny :-) I would prefer "Primary, Secondary, > and Tertiary". Then you can abbreviate them "PBL, SBL, and TBL" or "BL1, BL2, > and BL3" or whatever. Well, there is a real meaning between these grandma-mother-baby qualifiers, that would be lost with a BL1, BL2, BL3 scheme. You need only one grandma-BL (Jarl doesn't use the term, but in my mind it's the part responsible for "centralized distribution") to control a whole "piper program". You need many mother-BL's (one on each machine) for one grandma-BL. The same way, there are many baby-BL's for one mother-BL (many baby-BL's on each machine). If you don't like baby and grandma, what about super-BL (SBL), mother-BL (MBL) and child-BL (CBL)? BTW Jarl, your document really reflects what we agreed on friday! Jean-Marc -- Jean-Marc Valin Universite de Sherbrooke - Genie Electrique valj01@gel.usherb.ca From jarl at casema.net Mon Sep 11 11:22:57 2000 From: jarl at casema.net (jarl van katwijk) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] BL PL design documentation References: <39BC851A.45A9BE44@casema.net> <39BC171A.FE6CE4D2@bioinformatics.org> Message-ID: <39BCF8D1.BA296384@casema.net> > Question: What about the "Grandma BL"? We only use the BL and the babyBL. bye, jarl From rms at gnu.org Mon Sep 11 05:35:53 2000 From: rms at gnu.org (Richard Stallman) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] Re: GPL/LGPL issues In-Reply-To: <39BB26B9.B2F08439@hermes.usherb.ca> (message from Jean-Marc Valin on Sun, 10 Sep 2000 02:14:17 -0400) References: <39BB26B9.B2F08439@hermes.usherb.ca> Message-ID: <200009110935.DAA12998@wijiji.santafe.edu> The first one involves a library, libdata-flow which uses plugins by opening other shared libraries using dlopen(). Is that considered linking? In general, I believe it makes a combined program. However, in some trivial cases maybe one could make the argument that they are two separate programs that just happen to run in the same address space. If libdata-flow's license is the LGPL, would it be allowed to open a GPL plugin? Certainly. You can link LGPL-covered code with GPL-covered code in any fashion. would it be allow to open a closed-source plugin? (or both at the same time?) I don't want to speak about the categories of "open source" and "closed source", because that would be letting people make a mistake. I disagree fundamentally with the Open Source Movement, and if people think I support it, my work will all promote the wrong message. If I use their terminology in conversation, I will be encouraging that problem. See http://www.gnu.org/philosophy/free-software-for-freedom.html for more explanation. In the Free Software Movement we distinguish between free software and non-free; these are not the same categories as open or closed source, because the standards they adopted in 1998 are less strict than the ones we use. An open-source program is probably free software, but you can't be certain without checking the license. So if I used the terms "open source" and "closed source", I would also be talking about the wrong things. I can give you advice about free software based on the views of the Free Software Movement; if you would like advice from the Open Source Movement, you have to ask them. What I can say about the question of non-free plug-ins is that using the LGPL would definitely permit them. Is that a good idea? It depends on details of the situation, but in general I think it is a bad idea. It is better to insist that all add-ons must be free software. Otherwise we will face a constant struggle to develop free add-ons to compete with the non-free ones. Does the license of the program that links to libdata-flow change the answer to the previous questions? Yes. Our second issue has to do with CORBA. Is it allowed by the GPL to use CORBA to link a GPL'd program and a closed-source program? I can only speak about non-free programs, not about closed-source. However, I think that *in general* two modules that communicate through CORBA are separate programs, so that the license of each one does not cover the other. From chapmanb at arches.uga.edu Mon Sep 11 17:08:17 2000 From: chapmanb at arches.uga.edu (Brad Chapman) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] Baby-BL implementation In-Reply-To: <39BA93E9.507EDEC6@hermes.usherb.ca> Message-ID: <200009112108.RAA32974@archa10.cc.uga.edu> Hey all; Apologies in advance for being late on responding to all of this. I have just now come to some understanding of what is going on, and didn't want to respond when I was so confused. Sorry for being so slow on coming to grips with this! Jean-Marc wrote: > I've attached how I see the Baby-BL (which links to the PL) being > implemented. > Comments are welcome. This makes a lot of sense and seems very practical: 1. Fire up a CORBA server in the BBL and wait for calls from the MBL. 2. Call getOutput upon demand from the MBL to run a network (or subsection of a network). 3. Call fetchInput upon demand from the PL to get input from a particular node that may be remotely located. So, if I am with you so far, this just leads to a couple of questions: 1. It seems like this requires that the MBL be smart enough to know that it needs to split a network at display nodes, so that it can have access to the results that need to be displayed. So if we we had Jean-Marc's famous A->(B->C)->D network and say C were a viewer that needed to get results, then the MBL would have to know to split the network here and call getOutput on C (before calling on D), so that is could have the output for display. Is this right, or am I way off base? 2. I'm not sure how the Overflow *Probe will work under this system. Am I missing something basic? Otherwise it looks great. You and Jarl seem to really have a good system down here. Please let me know how you'd like me to help with the implementation details of the BBL stuff (if at all!) and I'll be more than happy to do some coding. Brad From chapmanb at arches.uga.edu Mon Sep 11 17:28:48 2000 From: chapmanb at arches.uga.edu (Brad Chapman) Date: Fri Feb 10 19:20:30 2006 Subject: [Pipet Devel] BL PL design documentation In-Reply-To: <39BC851A.45A9BE44@casema.net> Message-ID: <200009112128.RAA152736@archa11.cc.uga.edu> Jarl wrote: > As promised I've made a draft about the BL & PL design. Please > supply me with anything you can think of that I missed, so I can > finalize the document soon. This looks really nice Jarl. Thanks for writing this up so clearly and concisely. I just have one question about it all: You only men