ant-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Performance of fileset related operations with a large number of files
Date Fri, 20 Jan 2006 10:51:11 GMT
Stefan Bodewig wrote:
> On Thu, 19 Jan 2006, Paul Mackay <Paul.Mackay@nokia.com> wrote:
> 
>> Is there any way to avoid this high memory usage and time spent
>> building a list?
> 
> No.  And you'll see that Ant 1.7 is both a bit better and a bit worse
> than it used to be.  Directoryscanner has probably become a bit faster
> but at the same time we've broadened the concept of FileSets to
> ResourceCollections which means the copy task now works on more
> complex structures and even non-Files which probably leads to a
> further slowdown.
> 
>> Has there ever been any consideration of refactoring the way Ant
>> processes filesets and similar constructs such that each selected
>> file is processed once read in an iterative fashion, rather than
>> building a complete list and then processing?
> 
> Apart from cosmetics like printing the number of files to copy (before
> actually copying them) and backwards compatibility that Jeffrey
> mentioned this would also break optimizations in Move, which checks
> whether a fileset matches a whole directory tree and then simply moves
> the root of that tree instead of the individual files.  To do that
> Move has to complete the directory scans before it starts to move
> anything.
> 

Its interesting to note that even scp, my favourite dependency-driven 
copy command (I often use it locally for its logic) does build up a file 
list, so it has startup costs too.

Maybe a custom <dumbcopy> task could do optimal copying, one with its 
own file collection type and a runtime what would do bulk nio copying 
while still enumerating what other things to copy. It'd be hard to 
optimise as you get such different behaviour from different filesystems, 
hdds, network cables, etc.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@ant.apache.org
For additional commands, e-mail: dev-help@ant.apache.org


Mime
View raw message