ant-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Rice <rob...@windermere.com>
Subject Help getting a large number of files from an ftp server.
Date Mon, 02 Jun 2003 22:16:31 GMT
I've been working on an ant target that is required to get a number of 
files from an ftp server.  So far, I've been using the <ftp> task to do 
the work.  I have it operating on an includesfile, that describes the 
files to download.

This process seems to work rather well with a reasonably short list of 
files. 

A list of 10 files takes approx. 5 minutes.  A list of 35 files takes 
approx 10 minutes.  A list of 100 files takes approx 30 minutes. 

The problem is that my includesfile may have a couple thousand files to 
download.  If I send this many files to the <ftp> task, the server times 
out after about 4 hours, prior to downloading any files.  Here is the 
error.

BUILD FAILED
file:/home/robert/dvl/collector/NorthwestImageCollector.xml:76: Error 
while communicating with FTP server: 

Total time: 243 minutes 21 seconds

In all cases, success or otherwise, the task seems to hang for the 
majority of the processing time, prior to transferring any images.  The 
logged message is shown below.

ftp-images:
      [ftp] getting files

If the process makes it past this sticky point, then it rather quickly 
downloads the requested files with the following example logged messages:

      [ftp] transferring webphoto/bigphoto/96/23010396_04.jpg to 
/home/robert/dvl/collector/build/images/Northwest/webphoto/bigphoto/96/23010396_04.jpg
      [ftp] transferring webphoto/bigphoto/96/23010396_05.jpg to 
/home/robert/dvl/collector/build/images/Northwest/webphoto/bigphoto/96/23010396_05.jpg

Does anyone have suggestions on better ways to get a large number of files 
from an ftp server?

Thanks,
Robert Rice




Mime
View raw message