[Biodevelopers] XML for huge DB?

[Dan Bolser]

No, the problem is that a big results file can grab 50% of the 4GB
memory on the system. When I run 4 processes (and a file of this
size takes about 1 hour to process with XML::Simple) then as soon
as more that one process encounters a big file I am skuppered.

I am looking for a memory lite way of parsing the blast results
files from XML, I.E. one HST at a time with a print event
for each, rather than whole file at a time processing from
XML::Simple....

Dan.

Michael Gruenberger wrote:

>Hello,
>
>if you are on Unix, does 'nice' do what you are asking?!
>See:
>http://www.phys.ksu.edu/~esry/Computing/Nicing.html
>And: 
>man nice
>
>I don't know if it affects memory usage, but you can give your parsing
>process a lower priority so it shouldn't take your whole system down...
>
>Cheers,
>
>Michael
>On Thu, 2003-07-31 at 16:02, Dan Bolser wrote:
>  
>
>>Hello,
>>
>>How can I use XML efficiently to parse multiple blast results
>>files?
>>
>>I want to parse them on a multi processor environment, without
>>hitting the system memory limit.
>>
>>This is likely to happen, as big files take the most time, so the
>>processes tend to work on big files at the same time, leading
>>to a system memory outage....
>>
>>Cheers,
>>Dan.
>>
>>_______________________________________________
>>Biodevelopers mailing list
>>Biodevelopers at bioinformatics.org
>>https://bioinformatics.org/mailman/listinfo/biodevelopers
>>    
>>