hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10560) Update NativeS3FileSystem to issue copy commands for files with in a directory with a configurable number of threads
Date Thu, 07 May 2015 09:21:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532309#comment-14532309
] 

Hadoop QA commented on HADOOP-10560:
------------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch |   0m  0s | The patch command could not apply the patch during
dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12643032/HADOOP-10560-1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 305e473 |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/6517/console |


This message was automatically generated.

> Update NativeS3FileSystem to issue copy commands for files with in a directory with a
configurable number of threads
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-10560
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10560
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>            Reporter: Ted Malaska
>            Assignee: Ted Malaska
>            Priority: Minor
>              Labels: BB2015-05-TBR, performance
>         Attachments: HADOOP-10560-1.patch, HADOOP-10560.patch
>
>
> In NativeS3FileSystem if you do a copy of a directory it will copy all the files to the
new location, but it will do this with one thread. Code is below. This jira will allow a configurable
number of threads to be used to issue the copy commands to S3.
> do {
> PartialListing listing = store.list(srcKey, S3_MAX_LISTING_LENGTH, priorLastKey, true);
> for (FileMetadata file : listing.getFiles())
> { keysToDelete.add(file.getKey()); store.copy(file.getKey(), dstKey + file.getKey().substring(srcKey.length()));
}
> priorLastKey = listing.getPriorLastKey();
> } while (priorLastKey != null);



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message