hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
Date Wed, 15 May 2013 22:31:18 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Vinod Kumar Vavilapalli updated MAPREDUCE-5221:

    Status: Open  (was: Patch Available)

Thanks for taking this up, Tsuyoshi!

Comments on your patch:
 - In ReduceTask.getCombinerClass(), there is a lot of implicit type-casting. I am not sure
if this will even work. getCombinerClass(), the variable clazz pointing to shuffle-plugin
and ShuffleConsumerPlugin.Context all take in a Class<? extends Reducer> which won't
be true for the new combiner. Did you test this at all? Can you please run a job with a new
combiner with the patch? I doubt that it works.
 - Please fix the typos "Cominer" in the tests.
 - May be also add another simple test to verify if both new and old combiners are set.
 - Move this test code to the existing mapred.TestReduceTask.

> Reduce side Combiner is not used when using the new API
> -------------------------------------------------------
>                 Key: MAPREDUCE-5221
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.0.4-alpha
>            Reporter: Siddharth Seth
>            Assignee: Tsuyoshi OZAWA
>         Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch, MAPREDUCE-5221.3.patch
> If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this will silently
ignored on the reduce side since the reduce side usage is only aware of the old api combiner.
> This doesn't fail the job - since the new combiner key does not deprecate the old key.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message