hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepika Khera <Deepika.Kh...@avg.com>
Subject RE: Re: Hash Partitioner
Date Mon, 24 May 2010 22:32:52 GMT
Thanks for your response Eric.

I am using hadoop 0.20.2.

Here is what the hashCode() implementation looks like (I actually had the IDE generate it
for me)

Main key (for mapper & reducer):

public int hashCode() {
        int result = kVersion;
        result = 31 * result + (aKey != null ? aKey.hashCode() : 0);
        result = 31 * result + (gKey != null ? gKey.hashCode() : 0);
        result = 31 * result + (int) (date ^ (date >>> 32));
        result = 31 * result + (ma != null ? ma.hashCode() : 0);
        result = 31 * result + (cl != null ? cl.hashCode() : 0);
        return result;
    }
    
    
aKey : AKey class

    
    public int hashCode() {
        int result = kVersion;
        result = 31 * result + (v != null ? v.hashCode() : 0);
        result = 31 * result + (s != null ? s.hashCode() : 0);
        result = 31 * result + (o != null ? o.hashCode() : 0);
        result = 31 * result + (l != null ? l.hashCode() : 0);
        result = 31 * result + (e ? 1 : 0); //boolean
        result = 31 * result + (li ? 1 : 0); //boolean
        result = 31 * result + (aut ? 1 : 0); //boolean
        return result;
    }

When this happens, I do see the same values for the key. Also I am not using a grouping comparator.

I was wondering since the call to HashPartitioner.getPartition() is done from a map task,
several of which are running on different machines, is it possible that they get a different
hashcode and hence get different reducers assigned even when the key is the same.

Thanks,
Deepika



-----Original Message-----
From: Eric Sammer [mailto:esammer@cloudera.com] 
Sent: Monday, May 24, 2010 3:07 PM
To: general@hadoop.apache.org
Subject: Re: Hash Partitioner

Deepika:

That sounds very strange. Can you let us know what version of Hadoop
(e.g. Apache 0.20.x, CDH2, etc.) you're running and a bit more about
your hashCode() implementation? When this happens, do you see the same
values for the duplicate key? Did you also implement a grouping
comparator?

The hash partitioner is extremely simple. It basically does
key.hashCode() % numberOfReduces = partition number to which a key is
assigned. If one incorrectly implements a grouping comparator, it's
possible you could see odd behavior, though.

On Mon, May 24, 2010 at 5:35 PM, Deepika Khera <Deepika.Khera@avg.com> wrote:
> Hi,
>
> I am using a HashPartitioner on my key for a map reducer job.  I am wondering how sometimes
2 reducers end up getting the same key ? I have the hashCode method defined for my key.
>
> Also, I have speculative execution turned off for my jobs..
>
> Would appreciate any help.
>
> Thanks,
> Deepika
>



-- 
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com

Mime
View raw message