hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Didonna <m.didonn...@gmail.com>
Subject Re: Reducer getting key-value pairs in wrong order
Date Sat, 05 Feb 2011 09:16:03 GMT
On 02/04/2011 10:11 AM, Marco Didonna wrote:
> On 02/03/2011 07:02 PM, Harsh J wrote:
>> For a ValueGrouping comparator to work, your Partitioner must act in
>> tandem with it. I do not know if you have implemented a custom
>> hashCode() method for your Key class, but your partitioner should look
>> like:
> 
> Yes I did and it works like this return "leftElement.hashCode() +
> rightElement; "
> 
>>
>> return (key.getLeftElement().hashCode() & Integer.MAX_VALUE) % numPartitions;
>>
> 
> This was definitely a bug, the result is always the same though :(
> 
>> This will ensure that the to-be grouped data is actually partitioned
>> properly too.
>>
>> The actual sorting (which ought to occur for the full composite key
>> field-by-field, and is the only real 'sorter') would be handled by the
>> compare() call of your Writable, if you are using a
>> WritableComparable.
> 
> I am using a WritableComparable...here's PairOfStringInt
> https://gist.github.com/810905
> 
> Thanks again
> 
> 


I finally made it https://gist.github.com/809803 I use the
groupingComparator as job.setSortComparatorClass(GroupingComparator.class)

I still do not understand what was wrong with the old version of the
GroupingComparator and when the key are ordered according to the policy
encoded in GroupingComparator.

MD


Mime
View raw message