ant-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Rice <rob...@windermere.com>
Subject Re: Help getting a large number of files from an ftp server. (fwd)
Date Tue, 03 Jun 2003 01:02:24 GMT
Requirements:

I wish to download every file specified in a list.  Currently, this 
list is provided in a file.  Each line of that file defines a file to 
download.  Each file in the list is described as an absolute path from 
the ftp server root.  

Ideally, I also would like to have dependency checking, so as not to 
download files that may already be current.

Ideally also, any missing local subdirectories will be automatically 
created.  

Sample list:

webphoto/bigphoto/47/22085147_01.jpg
webphoto/bigphoto/18/22215818.jpg
webphoto/bigphoto/18/22215818_01.jpg
webphoto/bigphoto/55/22223155.jpg
webphoto/bigphoto/55/22223155_01.jpg

Sample task:

  <!-- ftp the images -->
  <target name="ftp-images" depends="changelog">
    <ftp action="get"
         server="${mls.server}"
         userid="${mls.username}"
         password="${mls.password}"
         verbose="yes"
         depends="yes"
         skipFailedTransfers="yes">
      <fileset dir="${build.images.dir}" 
               includesfile="${build.images.dir}/${build.images.changelog.file}">
      </fileset>
    </ftp>
  </target>

Bug 20103:

I am studying bug 20103, and it does look quite promising as a similar or 
exact problem I am up against.  I will study it further. 

Can you suggest a work around or another approach based on the 
requirements.

Thanks,
Robert Rice

On Tue, 3 Jun 2003, Antoine Levy-Lambert wrote:

> Could you describe in functional terms what you want to do, for instance
> which are the criteria met by the files you want to download, as opposed to
> the files you do not want to download ?
> See the bug number 20103
> http://nagoya.apache.org/bugzilla/show_bug.cgi?id=20103
> this one might be related to your problem or your problem exactly. It would
> mean that your problem boils down to a performance problem of <fileset/>
> rather than of <ftp/> per se.
> If you think this <guess/> is OK, you might want to add a comment describing
> your problem in the same bug report, and add yourself as CC on the bug
> report to see how resolution progresses.
> Cheers
> Antoine
> ----- Original Message -----
> From: "Robert Rice" <robert@windermere.com>
> To: <user@ant.apache.org>
> Sent: Tuesday, June 03, 2003 12:16 AM
> Subject: Help getting a large number of files from an ftp server.
> 
> 
> > I've been working on an ant target that is required to get a number of
> > files from an ftp server.  So far, I've been using the <ftp> task to do
> > the work.  I have it operating on an includesfile, that describes the
> > files to download.
> >
> > This process seems to work rather well with a reasonably short list of
> > files.
> >
> > A list of 10 files takes approx. 5 minutes.  A list of 35 files takes
> > approx 10 minutes.  A list of 100 files takes approx 30 minutes.
> >
> > The problem is that my includesfile may have a couple thousand files to
> > download.  If I send this many files to the <ftp> task, the server times
> > out after about 4 hours, prior to downloading any files.  Here is the
> > error.
> >
> > BUILD FAILED
> > file:/home/robert/dvl/collector/NorthwestImageCollector.xml:76: Error
> > while communicating with FTP server:
> >
> > Total time: 243 minutes 21 seconds
> >
> > In all cases, success or otherwise, the task seems to hang for the
> > majority of the processing time, prior to transferring any images.  The
> > logged message is shown below.
> >
> > ftp-images:
> >       [ftp] getting files
> >
> > If the process makes it past this sticky point, then it rather quickly
> > downloads the requested files with the following example logged messages:
> >
> >       [ftp] transferring webphoto/bigphoto/96/23010396_04.jpg to
> >
> /home/robert/dvl/collector/build/images/Northwest/webphoto/bigphoto/96/23010
> 396_04.jpg
> >       [ftp] transferring webphoto/bigphoto/96/23010396_05.jpg to
> >
> /home/robert/dvl/collector/build/images/Northwest/webphoto/bigphoto/96/23010
> 396_05.jpg
> >
> > Does anyone have suggestions on better ways to get a large number of files
> > from an ftp server?
> >
> > Thanks,
> > Robert Rice
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> > For additional commands, e-mail: user-help@ant.apache.org
> >
> >
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@ant.apache.org
> For additional commands, e-mail: user-help@ant.apache.org
> 



Mime
View raw message