[Pipet Devel] Open DX as a loci skeleton?

[Gary Van Domselaar]

Harry Mangalam wrote:
> 
> Hi Gary,
> > I mentioned IBM's Data Explorer back in the fall.  It is a very
> > sophisticated and powerful data visualization environment.
> 
> My apologies - I must have been asleep (or just ignoring everything that
> wasn't necessary to my continued survival at the time...)

Hmmm.. I can't find it on the list... might have just sent it directly
to Jeff... me bad!
 

> 
> > Considering that Loci's and OpenDX's architectures are so closely aligned,
> > and that we have yet to hammer out the details of the 'executive', I think
> > we would be wise to take a peek at their code.
> 
> So I'm not entirely in left field :) or if I am, at least Gary's standing
> next to me.

Hey, we're all in this game together!

> 
> What remains to be seen (or at least understood, from my end) is whether or
> how the servers can be tied into a distributed system that is as
> 'self-aware' as what has been proposed for Loci.  (He Wildly Waves the
> Not-A-CORBA-Expert flag)  I'm wondering if this awareness might be mediated
> by a CORBA layer - Gary - do you know if there is a CORBA component to the
> DX communication layer?  There doesn't appear to be from a cursory rip thru
> the docs, but even if there isn't, that might be the key component that
> Loci (or NCGR) could contribute.  I just posted a note to the DX list about
> this.

I haven't had the opportunity to examine their architecture in detail. 
My assumptions were that it doesn't use CORBA/XML, essentially because
(as Jeff had mentioned) Data Explorer was developed back in the mesozoic
era (~1991), when these open standards weren't around.  Loci and TOL
have invested heavily in these open standards, so IMHO the amount of
code reuse we could get out of OpenDX depends I think pretty heavily on
whether it has an open standard or proprietary data communication layer.
Regardless, the extensibility, modularity, generic data model, and
traffic scheduling issues are fundamentally important for Loci; and
implementing them properly is not trivial.  OpenDX I think would make an
excellent study-model for Loci.   
> 
> However, Doug Schmidt (who coincidentally works at UCI, a few minutes from
> me) has put together an Open Source Adaptive Communications Environment
> (ACE - http://www.cs.wustl.edu/~schmidt/ACE-overview.html ) that uses CORBA
> for creating distributed computing environments.  The combination of the
> two might be what's needed.

Thanks for the link. I glanced over it last nite, and it does look very
interesting (and complicated--my head was just swimming after reading
some of those docs!).   Reccommended reading for all!  I found this one
especially informative:


http://www.cs.wustl.edu/~schmidt/CACM-lessons.html

"Lessons Learned Building Reusable OO Frameworks for Distributed
Software"


Another fundamentally important issue for Loci is the capability to
handle enormous datasets efficiently. Bioinformatics is quickly reaching
the point where terabytes of data must be swung around between processes
in a distributed environment.  This raises issues of multicasting,
streaming, flexible reintegration of processed data, and a lot more
stuff that I don't even really know about yet.  How well do OpenDX and
ACE accomodate these issues?  I really don't know.  I'll explore OpenDX
and ACE, and get back to y'all on this issue when I have a get the
low-down :-)

Regards,

g.
-- 
                                  Gary Van Domselaar
                               gary at bioinformatics.org
                          http://www.bioinformatics.org/~gary
                ----------------------------------------------------
                           bioinformatics.org: The Open Lab
                            http://www.bioinformatics.org/