hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4824) Provide a mechanism for jobs to indicate they should not be recovered on restart
Date Wed, 28 Nov 2012 16:06:58 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505596#comment-13505596
] 

Harsh J commented on MAPREDUCE-4824:
------------------------------------

bq. I didn't add the property to mapred-default.xml, since it is a job-specific property and
these are generally not added there.

We do have several job-specific properties with proper defaults listed in that file. Unless
someone overrides them manually, how come there is harm in doing this, and must we remove
the ones already present?

The file just helps serve as a good doc. behind the config feature, cause otherwise there's
no doc reference to this in the patch.
                
> Provide a mechanism for jobs to indicate they should not be recovered on restart
> --------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-4824
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4824
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv1
>    Affects Versions: 1.1.0
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-4824.patch, MAPREDUCE-4824.patch
>
>
> Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on
jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there
is not applicable for MR1, since even if we only use the job-level part of the patch and add
a isRecoverySupported method to OutputCommitter, there is no way to use that information from
the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and
it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.)
> Instead, we can add a MR configuration property to say that a job is not recoverable,
and the JT could safely read this from the job conf.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message