[OpenMMS] db-loader

[Dan Bolser]

On 07/03/2008, Semin Lee <seminlee at gmail.com> wrote:
>
>  Dan Bolser wrote:
>  On 07/03/2008, Semin Lee <seminlee at gmail.com> wrote:
>
>
>  Hi Dan,
>
>  Just read your post on pdb mailing list. Although I haven't thoroughly
>  checked the difference between the original mapping file and the revised
>  one, it seems that the revising step is for revising fields' lengths or
>  other properties, not the structure of schema. Probably, that's why the
>  resulting schema all look the same. Anyway, I hope there is someone who
>  can tell us the logic behind this 'revising' stuff more clearly. (I am
>  sorry that I am not posting this on the mailing list. I'm using a mail
>  bucket service, so I can only receive the mails from the mailing list.)
>
>  Hey Semin, thanks for the reply!
>
> I found that some data for some fields contain 'garbage' values - so
> the schema (field types / lengths etc) are all skewed by these
> entries. This creates problems in MySQL, because some index columns
> are turned into BLOB datatypes... I am thinking that running the
> script on 10 or so 'good' structures should be enough to 'finalize'
> the schema, allowing us to identify bad structures when they fail to
> load into the given schema...
>
>  Now I see the reason db-loader keep changing some fields' data types.
> Finalizing the schema using 'good' structures is a good idea, but it is
> still possible to lose some information like maximum field's length, isn't
> it?

Yes. Lets say the somehow 'approved' field length is 255 characters,
and our 10 good structures only use 50 characters. We then cause
problems for all the other structures with a longer value in that
field. However, basing the field types on *every* structure means that
a tiny fraction of badly formatted structures can skew the entire
schema. Perhaps it isn't critical, but its just something that I was
thinking about with regard to using DB-Loader


>  BTW, do you run incremental updates of DB-loader?
>
>  No, I just populate the database from scratch every week. Maybe it's not
> ideal but very straightforward. :-)

We do exactly the same with OpenMMS ;-)


>  P.S. I hope you don't mind, I mailed your reply back to the OpenMMS
> mailing list.
>
>
>  I don't mind at all. Thank you for sending my reply to the mailing list.

Your welcome. There were some OpenMMS developers on the list, but they
are not very active.


Take care,

Dan.


>
>
>
>  Cheers,
>
> Semin
>
>
>
>
>
>


-- 
hello