hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-14757) S3AFileSystem.innerRename() to size metadatastore lists better
Date Fri, 11 Aug 2017 13:28:00 GMT
Steve Loughran created HADOOP-14757:

             Summary: S3AFileSystem.innerRename() to size metadatastore lists better
                 Key: HADOOP-14757
                 URL: https://issues.apache.org/jira/browse/HADOOP-14757
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: HADOOP-13345
            Reporter: Steve Loughran
            Priority: Minor
             Fix For: HADOOP-13345

In {{S3AFileSystem.innerRename()}}, various ArrayLists are created to track paths to update;
these are created with the default size. It could/should be possible to allocate better, so
avoid expensive array growth & copy operations while iterating through the list of entries.

# for a single file copy, sizes == 1
# for a recursive copy, the outcome of the first real LIST will either provide the actual
size, or, if the list == the max response, a very large minimum size.

For #2, we'd need to get the hint of iterable length rather than just iterate through...some
interface {{{IterableLength.expectedMinimumSize()}} could do that.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org

View raw message