hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ste...@apache.org
Subject [hadoop] branch trunk updated: HADOOP-16775. DistCp reuses the same temp file within the task for different files.
Date Thu, 02 Jan 2020 15:36:56 GMT
This is an automated email from the ASF dual-hosted git repository.

stevel pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/hadoop.git


The following commit(s) were added to refs/heads/trunk by this push:
     new b6dc00f  HADOOP-16775. DistCp reuses the same temp file within the task for different
files.
b6dc00f is described below

commit b6dc00f481189821e5d982083eba6d01f108b3de
Author: Steve Loughran <stevel@cloudera.com>
AuthorDate: Thu Jan 2 15:36:25 2020 +0000

    HADOOP-16775. DistCp reuses the same temp file within the task for different files.
    
    Contributed by Amir Shenavandeh.
    
    This avoids overwrite consistency issues with S3 and other stores -though
    given S3's copy operation is O(data), you are still best of using -direct
    when distcp-ing to it.
    
    Change-Id: I8dc9f048ad0cc57ff01543b849da1ce4eaadf8c3
---
 .../java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java  | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
index 4683cdd..4ed856f 100644
--- a/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
+++ b/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
@@ -230,7 +230,8 @@ public class RetriableFileCopyCommand extends RetriableCommand {
     Path root = target.equals(targetWorkPath) ? targetWorkPath.getParent()
         : targetWorkPath;
     Path tempFile = new Path(root, ".distcp.tmp." +
-        context.getTaskAttemptID().toString());
+        context.getTaskAttemptID().toString() +
+        "." + String.valueOf(System.currentTimeMillis()));
     LOG.info("Creating temp file: {}", tempFile);
     return tempFile;
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message