hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
Date Fri, 07 Jun 2013 08:51:22 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13677898#comment-13677898

Tsuyoshi OZAWA commented on MAPREDUCE-5221:

bq. 1. There are no codes to test with MiniMRYarnCluster with launching reduce-side combiner.
TestMRAppWithCombiner seems to be enough, but reduce-side combiners are actually not launched
because data size is too small to launch combiners.

I've checked code again, and I've found that this is not true for reducer-side combiner. Reducer-side
combiner is launched every time InMemoryMerger#merge is called - e.g. when ShuffleScheduler#copySucceeded
is called. Therefore, TestMRAppWithCombiner with new MapReduce API is enough to test in this

And, MAPREDUCE-5294 and MAPREDUCE-5295 are now blockers of this ticket.
> Reduce side Combiner is not used when using the new API
> -------------------------------------------------------
>                 Key: MAPREDUCE-5221
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.0.4-alpha
>            Reporter: Siddharth Seth
>            Assignee: Tsuyoshi OZAWA
>         Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch, MAPREDUCE-5221.3.patch
> If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this will silently
ignored on the reduce side since the reduce side usage is only aware of the old api combiner.
> This doesn't fail the job - since the new combiner key does not deprecate the old key.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message