hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5974) Allow specifying multiple MapOutputCollectors with fallback
Date Fri, 22 Aug 2014 15:10:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106926#comment-14106926

Hudson commented on MAPREDUCE-5974:

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1871 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1871/])
MAPREDUCE-5974. Allow specifying multiple MapOutputCollectors with fallback. (Todd Lipcon
via kasha) (kasha: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1619492)
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/apt/PluggableShuffleAndPluggableSort.apt.vm

> Allow specifying multiple MapOutputCollectors with fallback
> -----------------------------------------------------------
>                 Key: MAPREDUCE-5974
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5974
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: task
>    Affects Versions: 2.6.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 2.6.0
>         Attachments: mapreduce-5974.txt
> Currently we only allow specifying a single MapOutputCollector implementation class in
a job. It would be nice to allow a comma-separated list of classes: we should try each collector
implementation in the user-specified order until we find one that can be successfully instantiated
and initted.
> This is useful for cases where a particular optimized collector implementation cannot
operate on all key/value types, or requires native code. The cluster administrator can configure
the cluster to try to use the optimized collector and fall back to the default collector.

This message was sent by Atlassian JIRA

View raw message