hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13600) S3a rename() to copy files in a directory in parallel
Date Sat, 09 Sep 2017 17:52:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160037#comment-16160037
] 

ASF GitHub Bot commented on HADOOP-13600:
-----------------------------------------

Github user steveloughran commented on a diff in the pull request:

    https://github.com/apache/hadoop/pull/157#discussion_r137931623
  
    --- Diff: hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
---
    @@ -143,9 +150,11 @@
       private Listing listing;
       private long partSize;
       private boolean enableMultiObjectsDelete;
    -  private TransferManager transfers;
    +  private TransferManager uploads;
    --- End diff --
    
    leave the name of this alone unless really, really needed because (a) we may do other
things with it in future, (b) reduces the size of the diff hence compatibility with other
patches.


> S3a rename() to copy files in a directory in parallel
> -----------------------------------------------------
>
>                 Key: HADOOP-13600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13600
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.7.3
>            Reporter: Steve Loughran
>            Assignee: Sahil Takiar
>
> Currently a directory rename does a one-by-one copy, making the request O(files * data).
If the copy operations were launched in parallel, the duration of the copy may be reducable
to the duration of the longest copy. For a directory with many files, this will be significant



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message