hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-972) distcp can timeout during rename operation to s3
Date Sun, 18 Oct 2009 23:36:31 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12767161#action_12767161

Chris Douglas commented on MAPREDUCE-972:

I see. Extending the FileSystem API is a non-starter, so we can move on from that. Progress
threads in general are discouraged (e.g. HADOOP-5052).

If I understand your proposal, the progress thread would report starting from the first rename,
but stop after some configurable interval. In most cases, I'm not sure how this would differ
from simply setting the task timeout higher, since progress is reported between renames. Also,
this wouldn't help renames after the thread exits.

Would it be sufficient to add a call to progress() right before the rename (after the delete)?
In that case, setting the task timeout higher would extend the time allowed for each rename,
which is the right level of granularity, anyway. It won't do this automatically for s3 destinations,
but pushing that detail into distcp is not ideal, either. One could add a FilterFileSystem
that resets a persistent progress thread for each rename, manage all the signaling/locking
etc., but its behavior seems indistinguishable from this much simpler tweak. Would this be

> distcp can timeout during rename operation to s3
> ------------------------------------------------
>                 Key: MAPREDUCE-972
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-972
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: distcp
>    Affects Versions: 0.20.1
>            Reporter: Aaron Kimball
>            Assignee: Aaron Kimball
>         Attachments: MAPREDUCE-972.2.patch, MAPREDUCE-972.3.patch, MAPREDUCE-972.4.patch,
MAPREDUCE-972.5.patch, MAPREDUCE-972.patch
> rename() in S3 is implemented as copy + delete. The S3 copy operation can perform very
slowly, which may cause task timeout.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message