hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4824) Provide a mechanism for jobs to indicate they should not be recovered on restart
Date Tue, 27 Nov 2012 20:05:57 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504897#comment-13504897

Harsh J commented on MAPREDUCE-4824:


- The message below in the exception can be improved I feel. I think its better to say "Job
ID was not recovered since it disabled recovery-upon-restart (mapred.job.restart.recover set
to false).". Also, since this case is to be expected (non-default override), I think it ought
to be a simple INFO log, but I understand we need to throw an Exception to halt the loading
of the JIP.

+      if (recovered && !conf.getBoolean("mapred.job.restart.recover", true)) {
+        throw new IOException("Job " + jobId + " should not be recovered " +
+            "since mapred.job.restart.recover is set to false.");
+      }

- We could also add this property to mapred-default.xml and document it that way.

The test changes look good.
> Provide a mechanism for jobs to indicate they should not be recovered on restart
> --------------------------------------------------------------------------------
>                 Key: MAPREDUCE-4824
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4824
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv1
>    Affects Versions: 1.1.0
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: MAPREDUCE-4824.patch
> Some jobs (like Sqoop or HBase jobs) are not idempotent, so should not be recovered on
jobtracker restart. MAPREDUCE-2702 solves this problem for MR2, however the approach there
is not applicable for MR1, since even if we only use the job-level part of the patch and add
a isRecoverySupported method to OutputCommitter, there is no way to use that information from
the JT (which initiates recovery), since the JT does not instantiate OutputCommitters - and
it shouldn't since they are user-level code. (In MR2 it's OK since the MR AM calls the method.)
> Instead, we can add a MR configuration property to say that a job is not recoverable,
and the JT could safely read this from the job conf.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message