hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5812) Make task context available to OutputCommitter.isRecoverySupported()
Date Tue, 22 Apr 2014 14:04:19 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976798#comment-13976798

Jason Lowe commented on MAPREDUCE-5812:

Thanks for updating the patch, Mohammad.  I think the second version of the patch is closer
to what we want, as the most recent patch has the mapred OutputCommitter#isRecoverySupported
method receiving a mapreduce JobContext when all of the other methods receive a mapred JobContext.
 I think for consistency we should do the bridging functions as was done for the other mapred
OutputCommitter methods so the JobContext type is consistent across the methods that derived
types would override.  This means the MRAppMaster will need to create the appropriate job
context object as JobImpl does when the job initializes.

>  Make task context available to OutputCommitter.isRecoverySupported()
> ---------------------------------------------------------------------
>                 Key: MAPREDUCE-5812
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5812
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mr-am
>    Affects Versions: 2.3.0
>            Reporter: Mohammad Kamrul Islam
>            Assignee: Mohammad Kamrul Islam
>         Attachments: MAPREDUCE-5812.1.patch, MAPREDUCE-5812.2.patch, MAPREDUCE-5812.3.patch
> Background
> ==========
> The system like Hive provides its version of  OutputCommitter. The custom implementation
of isRecoverySupported() requires task context. From taskContext:getConfiguration(), hive
checks if  hive-defined specific property is set or not. Based on the property value, it returns
true or false. However, in the current OutputCommitter:isRecoverySupported(), there is no
way of getting task config. As a result, user can't  turn on/off the MRAM recovery feature.
> Proposed resolution:
> ===============
> 1. Pass Task Context into  isRecoverySupported() method.
> Pros: Easy and clean
> Cons: Possible backward compatibility issue due to aPI changes. (Is it true?)
> 2. Call outputCommitter.setupTask(taskContext) from MRAM: The new OutputCommitter will
store the context in the class level variable and use it from  isRecoverySupported() 
> Props: No API changes. No backward compatibility issue. This call can be made from MRAppMaster.getOutputCommitter()
method for old API case.
> Cons: Might not be very clean solution due to class level variable.
> Please give your comments.

This message was sent by Atlassian JIRA

View raw message