hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
Date Tue, 04 Oct 2011 15:26:34 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120212#comment-13120212
] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-2702:
----------------------------------------------------

Looked at the patch. Some comments:
 - I think AppAttemptId is part of the API one way or the other. If we are not making the
appAttemptId as part of the API explicitly, we should document recoverTask() about the Id
being passed as part of the context/config so that folks with their own OuputCommitters can
know.
 - We should probably drop the boolean return value also and rely explicitly on the exception,
it isn't adding any value anyways.
 - We need to document that if recoverTask() throws an exception, that task's output won't
be used in the new generation.
 - You missed changes to the old-api that you hinted at?
 - The method names got a little confusing with so many usages of 'temp', how about simply
calling them getJobDirName() and getTaskDirName()?

(Funny we never had a test-case for FileOutputCommitter, good test Sharad :) )
                
> [MR-279] OutputCommitter changes for MR Application Master recovery
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2702
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>            Priority: Blocker
>         Attachments: MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch,
mr2702_v4.patch
>
>
> In MR AM recovers from a crash, it only reruns the non completed tasks. The completed
tasks (along with their output, if any) needs to be recovered from the previous life. This
would require some changes in OutputCommitter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message