hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4824) Provide a mechanism for jobs to indicate they should not be recovered on restart
Date Wed, 28 Nov 2012 15:35:01 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tom White updated MAPREDUCE-4824:

    Attachment: MAPREDUCE-4824.patch

Thanks for the feedback. Here's an updated patch with the improved message.

I didn't add the property to mapred-default.xml, since it is a job-specific property and these
are generally not added there. There's no way to have true job-specific properties, since
if someone adds the property to the jobtracker's mapred-site.xml file then it will be picked
up. I'm not sure there's an easy way around this. 
> Provide a mechanism for jobs to indicate they should not be recovered on restart
> --------------------------------------------------------------------------------
>                 Key: MAPREDUCE-4824
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4824
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv1
>    Affects Versions: 1.1.0
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-4824.patch, MAPREDUCE-4824.patch
> Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on
jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there
is not applicable for MR1, since even if we only use the job-level part of the patch and add
a isRecoverySupported method to OutputCommitter, there is no way to use that information from
the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and
it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.)
> Instead, we can add a MR configuration property to say that a job is not recoverable,
and the JT could safely read this from the job conf.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message