[Pipet Devel] the empire strikes back (was: summary for web page and announcements)

J.W. Bizzaro bizzaro at geoserve.net
Tue Apr 25 01:35:50 EDT 2000


This is the kind of explanation we need for the summary if we are to go
forward with the extra communication layer.  We have to be able to say WHY the
DL should be separate from the UI.  If people on this list don't understand
why, you can bet that people visiting our web site won't understand. 
Honestly, I didn't understand your point of view and probably still don't.  I
only heard that I need to 'respect existing code'.  But from my understanding,
it was only the preservation of existing code that was determining the design.

> Jeff, I'm not trying to place any kind of blame, and I don't need to
> be patted on the back to feel good about myself--all I am saying is
> that we should respect existing code when making design decisions.
> Previous changes have required changing code, which is fine, because
> things are not perfect.  But tossing away code, as you are suggesting
> to do here, is just not smart.

I would never do such a thing without thought, but you keep characterizing my
suggestion as being whimsical.

I don't need to remind you how much GMS and Overflow code will be 'tossed
away' for the collaborative.  Jarl said that GMS will be radically changed,
probably more so than any other program.  And Overflow will become a library
and lose its GUI (over 1,000 LOC's).  GMS will also lose its GUI.  So, what
will Loci lose?  I know, it's easy for me to say these things, since it's not
my own code that is in question.

> We already are sorely in need of
> programmers, and throwing away functional code is not the way to make
> best use of our time, or encourage new people to join on.

much else to be done.  I was talking about the final state of VSh.

> I know that
> things change, but you have to understand how it would make you feel
> to have people talking about dumping something that you have put a lot
> of work into.

Probably a lot like Jean-Marc and Jarl feel ;-)

> In this case dumping code is the easiest decision to
> make, but not the best--we need to think up a solution which does not
> involve this.

I think the best decision to make is the one that you feel most comfortable
with.  You can tell that I am a perfectionist.  I tend to work and rework
everything until I like what I've done.  I personally want to feel right about
the design of VSh, especially since this is something that has consumed and
will continue to consume years of my life.  But the same goes for everyone
else.  Do you feel right about your design decision, Brad?

> I disagree. GMS is not the middle we were talking about before--the dl
> is. The middle has/will have a lot of functionality that is not
> present in the bl/gms:
> 1. Representing a gui as xml in both temporary and permanent storage.

Isn't this kept with the node and network information/XML?

> 2. Communicating with non-program/library elements, such as the local
> filesystem and databases.

What is to be kept in the local filesystem and databases?

> 3. Converting the xml format used to represent and store info for the
> gui into a format that can be easily read by a processing engine.

Hmmm.  I thought the PL's wrapping mechanism was going to do this, or maybe
the BL.  How much XML is stripped off before the BL and PL get the

> 4. Authenticating remote connections from other Vsh programs.

Yeah, but you and Jarl have plans for DL->BL authentication.  Are you talking
about this or a UI->DL authentication system?

 > Lumping this functionality into the front will make it *extremely
> unlikely* that anyone will ever bother to implement a different front
> end, as this is too much crap to deal with. Lumping this functionality
> into gms/overflow will make these programs bloated and make it
> difficult to do cool things like increase processing efficiency.

But I'm not convinced that something so close to the front as the DL is should
be manipulating the network structure and managing the filesystem.  I see
these as core operations.  This is why I suggested the DL be part of the BL.

>     The original design barely even took into account having a
> processing component since that was still waaaaaaay in the future.
> Thanks to Jarl we have that now, so we just need to modify the design
> to incorporate this.

But the PL (Overflow) handles processing.  Now you're saying Jarl's BL does. 
I'm very confused.

>     As far as I'm concerned we are dealing with more a point of view
> problem than an actual problem with the functional code. Can you point
> out specific problems in the current ui to dl communication which make
> it worth throwing out?

Because both UI->DL and DL->BL ...

    (1) Act like client->server
    (2) Pass XML
    (3) Need authentication to connect
    (4) Can connect across a network

and because there would be numerous advantages to combining UI and DL:

    (5) Speed: Direct Python communication rather than socket
    (6) Less processes running: 1 process rather than 2
    (7) Unified code base
    (8) One less authentication system needed (recall we would need 3)
    (9) Elimination of sockets: keep corba as the only protocol
    (10) No XML management (reading and writing) between the two

> > What we have now is...
> >
> >    [Loci front -> middle] -> VSh middle
> What do the []s mean? I just see us as having:

    [Loci front -> middle] == VSh front

>     loci front -> loci middle -> gms
> So what? We are dealing with two totally separate apis, and the
> middle communicates with them. Dealing with a user interface is a
> serious pain, you have to take into account that the user can
> connect, disconnect, delete, add, and otherwise do whatever they want
> whenever the want. Thus the api for communicating with a ui is going
> to be inherently huge and messy.

But from what I understand, and I think Jarl agrees with me, one DL represents
one user: 1UI:1DL and xDL:1BL.  So, then DL->BL already handles issues with
the user.

>     By contrast the api for communicating with the processing engine
> is nice and clean--give it some stuff to process, and get your results
> back out. Aaaah, so nice :)

Ummm.  What are you calling the 'processing engine'?  BL+PL or just PL?

>     The current middle/dl will do the translation between the ui api
> and the processing api and hopefully allow both to function at their
> best. For a program with proposed multiple user interfaces (an
> excellent idea), you need something to translate between the uis and
> the processing layer, and this is what the dl does.

But that is a 'core' function in VSh.  And I see the core as being BL+PL or
DL+BL+PL, not just the DL.

> The actual communication protocol isn't very important, it is the api
> that is being represented. Take a look at the communication protocol
> for front to middle communication and the corba interface for dl to
> bl communication. I don't see any kind of duplication between the two
> and the dl is the code that is doing the translation. If we want to
> try and connect a ui directly with a processing engine, then we will
> either get a huge crazy interface for the processing engine to have to
> deal with, or a ui that just mimics calls directly to the processing
> engine.

Well, it sounds to me like the DL is really inseparable from the BL+PL.  So,
we shouldn't have xDL:1BL as Jarl suggested; we should have xUI:1DL and

>     The Overflow gui is a great example of making direct processing to
> gui interface seen more natural. The gui is beautiful and (IMO) quite
> intuitive to use, but in its underlying design is still guiizing calls
> to functions. If you take a look at the menus they have tons of
> selections, all corresponding to individual functions/calls, and their
> xml design is set up to mimic a C program (as Jean-Marc was describing
> earlier). However, this seems like it will be hard to expand to
> include an arbitrary number of programs and be a "generalized"
> application wrapper. I think the dl can make this transition feasible
> (hopefully :).

The PL (Overflow) will be both a low-level development system and a wrapper. 
It will wrap command-line programs the way it does now (maybe with some

> I just don't understand the horrendous consequences you seem to see if
> adding an extra communication layer. Maybe you could describe the
> specific problems you see.

Alright, I'll ask you this: Would you have come up with the extra
comminication layer if you sat down and designed VSh from scratch?

> > We COULD also consider combining Loci's middle with the VSh BL (using
> > C-Python
> > bindings).
> This is no different than communicating through corba, it is just a
> different way to do it. CORBA is much more flexible since we can
> change the interface while we are doing things, while wrappers will
> make changing things a whole lot harder.

You know, the 3 of us don't have a consistent view on this.  I'm sure Jarl
doesn't completely agree with you or I either.  And we just can't proceed
until we all know EXACTLY what each part does.  I honestly don't know.  I
can't even figure out what will hold the network structure.  Last time I asked
Jarl, he didn't seem sure either.  And I really want to know how many UI's per
DL, how many DL's per BL, and how many BL's per PL.  I'm in the dark trying to
help answer these questions, but my answers are coming back as 'unacceptable'
and 'you need to respect my code'.

> I wasn't saying that I want complete design control--I know that I am
> not a good designer, which is why I'm working on problems with other
> people. I just want design decisions to be made respecting working
> contributed code. Since we now have actual code, we can't afford to
> make designs as if everything is vaporware, and need to work around
> what we already have. I would say this even if it wasn't my own code
> that we are talking about :-)

Pardon me for the sarcasm, but should we then patch the GMS and Overflow GUI's
into VSh somewhere? ;-)  You haven't mentioned that at all.  Brad, I think I
abhor a patch-work, hodge-podge application as much as you hate throwing away

                      |           J.W. Bizzaro           |
                      |                                  |
                      | http://bioinformatics.org/~jeff/ |
                      |                                  |
                      |        BIOINFORMATICS.ORG        |
                      |           The Open Lab           |
                      |                                  |
                      |    http://bioinformatics.org/    |

More information about the Pipet-Devel mailing list