hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5051) Combiner not used when NUM_REDUCES=0
Date Fri, 08 Mar 2013 19:04:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13597403#comment-13597403

Robert Joseph Evans commented on MAPREDUCE-5051:

In trunk it is the MergeManagerImpl that calls the combiner


In branch-1 MapTask.java itself handles the combiner code. You can look at Task.java for some
of the implementations of it.
> Combiner not used when NUM_REDUCES=0
> ------------------------------------
>                 Key: MAPREDUCE-5051
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5051
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv1
>    Affects Versions: 2.0.2-alpha
>         Environment: CDH4.1.2 MR1
>            Reporter: Damien Hardy
> We have a M/R job that use Mapper + Combiner but have nothing to do in Reducer :
> Bulk indexing of HBase data in ElasticSearch,
> Map output is K / V : #bulk / json_data_to_be_indexed.
> So job is launched maps work, combiners index and a reducer is created for nothing (sometimes
waiting for other M/R job to free a tasktracker slot for reducer cf. MAPREDUCE-5019 )
> When we put ```job.setNumReduceTasks(0);``` in our job .run(), mapper are started but
combiner are not used.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message