Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Message-ID: <1688317156.1255970879597.JavaMail.jira@brutus>
Date: Mon, 19 Oct 2009 16:47:59 +0000 (UTC)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Subject: [jira] Commented: (MAPREDUCE-972) distcp can timeout during rename
 operation to s3
In-Reply-To: <431777488.1252632958336.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767377#action_12767377 ] 

Doug Cutting commented on MAPREDUCE-972:
----------------------------------------

> Extending the FileSystem API is a non-starter [ ... ]

Isn't the FileSystem API mid-rewrite right now in HADOOP-6223?  So now might actually be the rare time to consider something like this.  It's unfortunate that Rename.Options is an Enum, so it'd be hard to add a progress function there without changing that.  Perhaps Rename.Options.OVERWRITE could still be a constant, but Rename.Options#createProgress(Progressible) could return a subclass of Rename.Options that wraps a Progressible or somesuch.  I don't mean to push this approach, rather just to question whether it should be ruled out completely.  If it seems reasonable for file rename implementations to take a long time, then adding a progress callback might be a reasonable approach.


> distcp can timeout during rename operation to s3
> ------------------------------------------------
>
>                 Key: MAPREDUCE-972
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-972
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: distcp
>    Affects Versions: 0.20.1
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: MAPREDUCE-972.2.patch, MAPREDUCE-972.3.patch, MAPREDUCE-972.4.patch, MAPREDUCE-972.5.patch, MAPREDUCE-972.patch
>
>
> rename() in S3 is implemented as copy + delete. The S3 copy operation can perform very slowly, which may cause task timeout.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.