hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsuyoshi OZAWA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
Date Sat, 22 Jun 2013 15:18:22 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13691165#comment-13691165
] 

Tsuyoshi OZAWA commented on MAPREDUCE-5221:
-------------------------------------------

[~kkambatl], I understood what you mean correctly. I explain why I choose the current design.

bq. I thought all of MAPREDUCE-5221, MAPREDUCE-5294 and MAPREDUCE-5295 are related and motivated
by the same bug (from the description) that reduce-side combiner is not called using the new
API. 

That's right.

bq. IIUC, we could just use an if condition similar to the one in my previous comment, and
that should solve the issue and we don't need the other two JIRAs. No? Am I missing something?

No, there are a missing point.
The sequence of running combiner is different between old API and new API.  Task#CombinerRunner
class abstracts it, so I choosed to reuse it in MAPREDUCE-5294. Do you think we don't use
it?

The patch for MAPREDUCE-5221 and MAPREDUCE-5295 just removes needless arguments or adds tests.
Should we merge these patches into one patch?
                
> Reduce side Combiner is not used when using the new API
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-5221
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 2.0.4-alpha
>            Reporter: Siddharth Seth
>            Assignee: Tsuyoshi OZAWA
>         Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch, MAPREDUCE-5221.3.patch,
MAPREDUCE-5221.4.patch
>
>
> If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this will silently
ignored on the reduce side since the reduce side usage is only aware of the old api combiner.
> This doesn't fail the job - since the new combiner key does not deprecate the old key.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message