hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Partitioner vs GroupComparator
Date Fri, 23 Aug 2013 16:05:55 GMT
The partitioner runs on the map-end. It assigns a partition ID
(reducer ID) to each key.
The grouping comparator runs on the reduce-end. It helps reducers,
which read off a merge-sorted single file, to understand how to break
the sequential file into reduce calls of <key, values[]>.

Typically one never overrides the GroupingComparator, and it is
usually the same as the SortComparator. But if you wish to do things
such as Secondary Sort, then overriding this comes useful - cause you
may want to sort over two parts of a key object, but only group by one
part, etc..

On Fri, Aug 23, 2013 at 8:49 PM, Eugene Morozov
<emorozov@griddynamics.com> wrote:
> Hello,
> I have two different types of keys emerged from Map and processed by Reduce.
> These keys have some part in common. And I'd like to have similar keys in
> one reducer. For that purpose I used Partitioner and partition everything
> gets in by this common part. It seems to be fine, but MRUnit seems doesn't
> know anything about Partitioners. So, here is where GroupComparator comes
> into play. It seems that MRUnit well aware of the guy, but it surprises me:
> it looks like Partitioner and GroupComparator are actually doing exactly
> same - they both somehow group keys to have them in one reducer.
> Could you shed some light on it, please.
> --

Harsh J

View raw message