hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
Date Fri, 21 Aug 2015 21:31:46 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14707498#comment-14707498
] 

Chris Douglas commented on MAPREDUCE-6434:
------------------------------------------

Agreed, the NPE is usually not a problem since the default should be defined in mapred-defaults,
though {{JobContextImpl::getReducerClass}} can return null. At least two cases shouldn't cause
a problem for map-only jobs:
# The base {{mapreduce.Reducer}} is {{\@Checkpointable}}, so it would instantiate a {{PartialFileOutputCommitter}}
# A {{Reducer}} in the config shouldn't cause a map-only job to fail if it's not on the classpath
(this may not be true in the current code, but this shouldn't add another case)

We also don't want to do anything surprising for setup/cleanup tasks.

> Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
> -------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6434
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Augusto Souza
>            Assignee: Augusto Souza
>         Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, MAPREDUCE-6434.003.patch,
MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch, MAPREDUCE-6434.006.patch
>
>
> Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable
now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed
or accepted by Jenkins CI scripts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message