hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
Date Fri, 21 Aug 2015 00:17:46 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14706018#comment-14706018
] 

Chris Douglas commented on MAPREDUCE-6434:
------------------------------------------

Thanks for updating the patch, [~augustorsouza]. Could you check this change?
{noformat}
-      committer = new FileOutputCommitter(output, context);
+      try {
+        if (context.getConfiguration().getBoolean(MRJobConfig.TASK_PREEMPTION,
+            false)
+            && context.getReducerClass()
+                .isAnnotationPresent(Checkpointable.class)) {
+          committer = new PartialFileOutputCommitter(output, context);
+        } else {
+          committer = new FileOutputCommitter(output, context);
+        }
+      } catch (ClassNotFoundException c) {
+        throw new RuntimeException(
+            "Internal error: reducer class is not defined ", c);
+      }
{noformat}

Since preemption in MAPREDUCE-5269 only supports reduce tasks, even if preemption is enabled
for map-only jobs, the reduce class can be undefined.

> Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
> -------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6434
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Augusto Souza
>            Assignee: Augusto Souza
>         Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, MAPREDUCE-6434.003.patch,
MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch
>
>
> Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable
now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed
or accepted by Jenkins CI scripts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message