Mailing-List: contact issues-help@commons.apache.org; run by ezmlm
Precedence: bulk
Reply-To: issues@commons.apache.org
Date: Fri, 6 May 2011 00:20:03 +0000 (UTC)
From: "Sebb (JIRA)" <jira@apache.org>
To: issues@commons.apache.org
Message-ID: 
 <232190989.26746.1304641203182.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1437856922.24418.1304595843118.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (IO-271) FileUtils.copyDirectory should be able
 to handle arbitrary number of files
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/IO-271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029669#comment-13029669 ] 

Sebb commented on IO-271:
-------------------------

Using list() instead of listFiles() would be possible, but would only double the size of a directory that could be processed.
The only way to truly fix the problem would be to use a method that provided access to the file names one by one, but there does not appear to be a method to do this.

AFAICT FileUtils does not override anything - anyway, why would it be necessary to delay updating the mod. date on the target file?

Personally, I don't think this is worth implementing. Users can always implement their own filtering to split the transfer into chunks. Or just make sure that directories don't contain so many files - this is likely to cause problems elsewhere as well.

> FileUtils.copyDirectory should be able to handle arbitrary number of files
> --------------------------------------------------------------------------
>
>                 Key: IO-271
>                 URL: https://issues.apache.org/jira/browse/IO-271
>             Project: Commons IO
>          Issue Type: Improvement
>          Components: Utilities
>    Affects Versions: 2.0.1
>            Reporter: Stephen Kestle
>
> File.listFiles() uses up to a bit over 2 times as much memory as File.list().  The latter should be used in doCopyDirectory where there is no filter specified.
> This memory usage is a problem when copying directories with hundreds of thousands of files.
> I was also thinking of the option of implementing a file filter (that could be composed with the inputted filter) that would batch the file copy operation; copy the first 10000 (that match), then the next 10000 etc etc.
> Because of the lack of ordering consistency (between runs) of File.listFiles(), there would need to be a final file filter that would accept files that have not successfully been copied.
> I'm primarily concerned about copying into an empty directory (I validate this beforehand), but for general operation where it's a merge, the modification date re-writing should only be done in the final run of copies so that while batching occurs (and indeed the final "missed" filtering) files do not get copied if they have been modified after the start time. (I presume that I'm reading FileUtils correctly in that it overrides files...)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira