hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stan Rosenberg <srosenb...@proclivitysystems.com>
Subject WritableComparable and the case of duplicate keys in the reducer
Date Sat, 13 Aug 2011 15:14:31 GMT
Hi All,

Here is what's happening.  I have implemented my own WritableComparable keys
and values.
Inside a reducer I am seeing 'reduce'  being invoked with the "same" key
I have checked that context.getKeyComparator() and
context.getSortComparator() are both WritableComparator which
indicates that 'compareTo' method of my key should be called when doing
reduce-side merge.

Indeed, inside the 'reduce' method I captured both key instances and did the
following checks:


In both calls, the result is '0', confirming that key1 and key2 are

So, what is going on?

Note that key1 and key2 come from different mappers but they should have
been collapsed in the reducer since
they are both equal according to WritableComparator.  Also note that key1
and key2 are not bitwise equivalent, but
that shouldn't matter, or should it?

Many thanks in advance!


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message