hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1574) Combiners should implement a specialized "Combiner" interface, not the generic "Reducer" interface
Date Mon, 15 Mar 2010 04:45:27 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12845180#action_12845180

Tom White commented on MAPREDUCE-1574:

Creating a Combiner class and _adding_ a setCombinerClass(Class<? extends Combiner>)
method, while retaining the Reducer one would not be incompatible. However, I'm  not convinced
it would make things clearer, as you would still be able to use a Reducer as a combiner -
and the conditions for this would still need explaining (only in the case that the types match,
and when it makes sense semantically for a reducer to be a combiner).

Perhaps we should be aiming for better diagnostics instead. We could improve the error message
to explain that the input and output types must match. This could be done at the point when
the framework receives a combiner  output key-value pair (or it might even be possible using
an approach like MAPREDUCE-1411).

> Combiners should implement a specialized "Combiner" interface, not the generic "Reducer"
> --------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-1574
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1574
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.20.1, 0.20.2, 0.20.3
>            Reporter: Danny Leshem
>            Priority: Minor
> I just spent 30 minutes trying to figure out why my job throws "java.io.IOException:
wrong key class" when I pass my Reducer class to Job.setCombinerClass. Finally, I understood
that a Reducer can act as Combiner only if its output key/value are the same as its input
> So yes, this is documented. But you can make life easier for users by defining a Combiner
interface (that Job.setCombinerClass will accept) to force this at compile time. The new interface
should implement the Reducer interface and specialize it (is it even possible with generics?).
Alternatively, you can call this interface "SimpleReducer".
> If the generics-trick suggested above is impossible to implement, for the (common?) case
of having the same class acting as Combiner and Reducer you can do one of either:
> 1) Thin Combiner implementation that wraps a given Reducer.
> 2) Add a new method, say Job.setCombinerClassToReducer (that accepts a Reducer), acting
similarly to the new Job.setCombinerClass - but here the name should alert the user she's
doing something special.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message