[Bioclusters] what are people using to mirror large ftp repositories

[Jeremy Mann]

> Hi folks,
> 
> I've turned a bunch of Seagate 160gb IDE disks into a large software 
> RAID5 volume and am trying to mirror the raw fasta data from 
> ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/ for use on a personal 
> development project.
> 
> The 'wget' utility (Redhat 9 on a dual PIII system w/ ext3 filesystem) 
> is bombing out on a few of the remote files which even when compressed 
> are greater than 2gb in size.
> 
> My kernel and ext3 filesystem support large filesizes but 'wget' or my 
> shell seem to have issues.

Ah yes, I spent a few minutes on getting ours to work. ncftpget is what 
you want to use. Support +2gig files. Command usage example:

ncftpget -d download.log ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/igSeqNt.Z
ncftpget -d download.log ftp://ftp.ncbi.nlm.nih.gov/blast/db/FASTA/igSeqProt.Z



-- 
Jeremy Mann
jeremy@biochem.uthscsa.edu
University of Texas Health Science Center
Bioinformatics Core Facility
(210) 567-2672