hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Lukavsk√Ĺ <jan.lukav...@firma.seznam.cz>
Subject Re: Partitioner vs GroupComparator
Date Fri, 23 Aug 2013 16:22:04 GMT
Hi all,

when speaking about this, has anyone ever measured how much more data 
needs to be transferred over the network when using GroupingComparator 
the way Harsh suggests? What do I mean, when you use the 
GroupingComparator, it hides you the real key that you have emitted from 
Mapper. You just see the first key in the reduce group and any data that 
was carried in the key needs to be duplicated in the value in order to 
be accessible on the reduce end.

Let's say you have key consisting of two parts (base, extension), you 
partition by the 'base' part and use GroupingComparator to group keys 
with the same base part. Than you have no other chance than to emit from 
Mapper something like this - (key: (base, extension), value: extension), 
which means the 'extension' part is duplicated in the data, that has to 
be transferred over the network. This overhead can be diminished by 
using compression between map and reduce side, but I believe that in 
some cases this can be significant.

It would be nice if the API allowed to access the 'real' key for each 
value, not only the first key of the reduce group. The only way to get 
rid of this overhead now is by not using the GroupingComparator and 
instead store some internal state in the Reducer class, that is 
persisted across mutliple calls to reduce() method, which in my opinion 
makes using GroupingComparator this way less 'preferred' way of doing 
secondary sort.

Does anyone have any experience with this overhead?


On 08/23/2013 06:05 PM, Harsh J wrote:
> The partitioner runs on the map-end. It assigns a partition ID
> (reducer ID) to each key.
> The grouping comparator runs on the reduce-end. It helps reducers,
> which read off a merge-sorted single file, to understand how to break
> the sequential file into reduce calls of <key, values[]>.
> Typically one never overrides the GroupingComparator, and it is
> usually the same as the SortComparator. But if you wish to do things
> such as Secondary Sort, then overriding this comes useful - cause you
> may want to sort over two parts of a key object, but only group by one
> part, etc..
> On Fri, Aug 23, 2013 at 8:49 PM, Eugene Morozov
> <emorozov@griddynamics.com> wrote:
>> Hello,
>> I have two different types of keys emerged from Map and processed by Reduce.
>> These keys have some part in common. And I'd like to have similar keys in
>> one reducer. For that purpose I used Partitioner and partition everything
>> gets in by this common part. It seems to be fine, but MRUnit seems doesn't
>> know anything about Partitioners. So, here is where GroupComparator comes
>> into play. It seems that MRUnit well aware of the guy, but it surprises me:
>> it looks like Partitioner and GroupComparator are actually doing exactly
>> same - they both somehow group keys to have them in one reducer.
>> Could you shed some light on it, please.
>> --

View raw message