hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
Date Tue, 04 Oct 2011 07:15:34 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119935#comment-13119935
] 

Arun C Murthy commented on MAPREDUCE-2702:
------------------------------------------

Sharad, apologies for not getting around to this, but I finally spent sufficient time on this
to do this important patch justice.

Overall, it looks great! Some minor nits:

# How about uniting isRecoverySupported and recoverTask into a single api: 'boolean recoverTask'.
So, an OutputCommitter can return false from 'recoverTask' to indicate it doesn't support
it.
# I'm a little concerned about introducing the notion of 'appAttemptId' as an integer to recoverTask.
This is probably one of the very few, if not the only place, where YARN leaks into 'classic'
MapReduce apis - particularly since these are user-facing apis. Maybe we can just pull it
out of TaskAttemptContext.configuration?
# Other minor, cosmetic suggestions for function names:
## getJobTempDir -> getJobTempDirName
## getTaskTempDir -> getTaskTempDirName
## moveOutputs -> moveJobOutputs

Thoughts?
                
> [MR-279] OutputCommitter changes for MR Application Master recovery
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2702
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>            Priority: Blocker
>         Attachments: mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch
>
>
> In MR AM recovers from a crash, it only reruns the non completed tasks. The completed
tasks (along with their output, if any) needs to be recovered from the previous life. This
would require some changes in OutputCommitter.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message