[Bioclusters] what are people using to mirror large ftp repositories with 2gb+ files?

[Fernan Aguero]

+----[ Chris Dagdigian <dag@sonsorol.org> (10.Sep.2003 17:28):
|
| My kernel and ext3 filesystem support large filesizes but 'wget' or my 
| shell seem to have issues.

I don't think the shell has anything to do here. 

| I've recompiled the wget .src.rpm with the usual compiler flags to add 
| large file support and wget _seems_ to be working but I don't really 
| trust it as it is reporting negative filesizes like this now:
| 
| > RETR nt.Z...done
| > Length: -1,668,277,957 [-1,705,001,669 to go]

perhaps the problem is with the ftp daemon at the server
side? the server should also have large file support and
have the daemon compiled with large file support enabled.

But, then I suppose that then others should be suffering the same
problem ...

I see that has already come up in the list
http://bioinformatics.org/pipermail/bioclusters/2003-January/000667.html

are you sure that there are no weird things going on
between your choice of kernel, filesystem, etc?

| What are others doing? Would 'curl' be better? Any recommendations would 
| be appreciated.

Long ago, when the linux 2.4 kernel and the large file
support were still in alpha testing, I smashed my head
several times against this problem. That was the main reason
behind my move to a BSD Unix system (actually FreeBSD).
The BSDs have had this support for a long while, which means
it is stable, and free of troubles.
You can compile the ncbi-toolkit on FreeBSD from the ports
collection (http://www.freebsd.org/ports), and use formatdb
without problem on large files. FTP, wget and friends do not
have problems either.

I know that switching to another OS is not a useful
advice. At least not in the short term. But it was for me,
and I have never had the need to go back to Linux, even when
I just have linux binaries and no access to sources for
compiling (FreeBSD can run linux binaries transparently).

Hope this helps,

Fernan

| 
| -Chris
| 
| 
| -- 
| Chris Dagdigian, <dag@sonsorol.org>
| BioTeam Inc. - Independent Bio-IT & Informatics consulting
| Office: 617-666-6454, Mobile: 617-877-5498, Fax: 425-699-0193
| PGP KeyID: 83D4310E iChat/AIM: 'bioteamdag' Web: http://bioteam.net
| 
| _______________________________________________
| Bioclusters maillist  -  Bioclusters@bioinformatics.org
| https://bioinformatics.org/mailman/listinfo/bioclusters
| 
|
+----]

-- 
F e r n a n   A g u e r o
http://genoma.unsam.edu.ar/~fernan