[Pipet Devel] Loci and data storage

[Gary Van Domselaar]

"J.W. Bizzaro" wrote:
> 
> Gary Van Domselaar wrote:
> >
> > I dont think this aspect of the loci core has been very thoroughly
> > addressed.  Does anyone have any ideas on how we might implement data
> > storage for Loci?
> 
> In early conversations, we realized the need to split the data into two basic
> types according to how they would be managed:
> 
>   (1) Data kept (as XML?) on the filesystem: mostly for storage; data
>       are not being passed via the Loci system
> 
>   (2) Data kept as (CORBA) objects: data that are being passed via Loci


That makes sense.


> 
> Alan then proposed the concept of pluggable/modular 'data managers'.  A DM would
> manage data of any specific type and is pretty much synonymous to what I have
> been calling a 'translator', which converts data from one format to another,
> plus the underlying infrastructure (what will actually be handled by CORBA).
> Here is an excerpt from Alan's e-mail on June 1 (this is in the archive; GST
> refers to General Sequence Toolkit):

Yes, I remember reading this.  My understanding was that the DM was
designed to extract data from existing databases, then convert the
information to a format appropriate for further analysis. I wasnt sure
that the concept had been extended to inputing (and maintaining) data
from a new or existing database.

_________
<snip>
> 
>    2) Intranet Oracle or AceDB database DM would consist of a local
>       plug-in (as well as a server for the plug-in possibly). The plug-in
>       would handle the network transparency as well as wrapping/converting
>       the data to xml.
</snip>

Here Alan mentions a plug-in interface to oracle and AceDB, but doesnt
really address the issue of data storage vs data extraction.

> > I'm no database expert, so I'm a little hesitant to
> > suggest how we should go about it, but it does seem important to me that
> > loci should be able to store analysis results in a relational database
> > (Informax's VectorNTI uses a relational database to store its data).
> 
> Hmmm.  That is providing (1) there are numerous analysis results to be stored
> and 'related', and (2) the user needs to store the results this way.  Are you
> suggesting this as an option or as a standard way of storing everything?

I think it depends on the nature of the data processing and analysis
that the end-user wishes to perform. If someone just wants an answer and
a pretty picture, then there is nothing to gain by storing the result in
a relational database. If, however, the researcher is working on a large
project, they may very well want to have the results stored in a
database, so that they can later query the database for interesting
patterns, irregularities, etc. So I guess it would be an option, just
one that I think is very important.


> > > This
> > would facilitate the construction of customized, sharable databases.  In
> > keeping with Loci's philosophy of not adopting specific data formats, it
> > seems to me that Loci should probably not adopt a single database, but
> > rather have the capability to interface with any of the popular
> > databases, such as oracle, sql, mysql, etc.  PHP has this capability, and
> > is one of the biggest reasons why it is so successful.
> 
> Right.  I think that is exactly what Alan was suggesting with the data manager
> proposal.  

So then, for clarity, if Loci were to allow for database storage of
analysis results, it would be implemented as a DM output option, with
the various available database interfaces as DM-plug-ins?


Regards

--gary 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Gary Van Domselaar		gvd at redpoll.pharmacy.ualberta.ca
Faculty of Pharmacy 		Phone: (780) 492-4493
University of Alberta		FAX:   (780) 492-5305
Edmonton, Alberta, Canada       http://redpoll.pharmacy.ualberta.ca/~gvd