Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of william.kinney@gmail.com
 designates 74.125.82.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CACvGQ9r6kQe76rnAGc-CftHSdMMPMciGZFjdDQb6EUbRr3aDXg@mail.gmail.com>
References: 
 <CAC_mgs+0dVGSknDkh76Xqxs_JQCn2=pEnhn1Ja0NYVsZn5AU2w@mail.gmail.com>
	<CACvGQ9rAZ7L5KKs_UC++0M3HYH-X7UY0ESgB=itUAAP4aVWkFA@mail.gmail.com>
	<CAN9z+orQtnZhMYnERx-+UMSWHD1fnaK8-13eYwn3h8QoyrOGFw@mail.gmail.com>
	<CACvGQ9r6kQe76rnAGc-CftHSdMMPMciGZFjdDQb6EUbRr3aDXg@mail.gmail.com>
Date: Tue, 10 Jan 2012 18:15:28 -0500
Message-ID: 
 <CACvGQ9osZQni0TN0W+wfB6LiocLsegr2xa+8XFRfxiG_8gU7Dg@mail.gmail.com>
Subject: Re: WritableComparable and the case of duplicate keys in the reducer
From: William Kinney <william.kinney@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0016e6de16b1dd768904b634b021

--0016e6de16b1dd768904b634b021
Content-Type: text/plain; charset=ISO-8859-1

Naturally after I send that email I find that I am wrong. I was also using
an enum field, which was the culprit.

On Tue, Jan 10, 2012 at 6:13 PM, William Kinney <william.kinney@gmail.com>wrote:

> I'm (unfortunately) aware of this and this isn't the issue. My key object
> contains only long, int and String values.
>
> The job map output is consistent, but the reduce input groups and values
> for the key vary from one job to the next on the same input. It's like it
> isn't properly comparing and partitioning the keys.
>
> I have properly implemented a hashCode(), equals() and the
> WritableComparable methods.
>
> Also not surprisingly when I use 1 reduce task, the output is correct.
>
>
> On Tue, Jan 10, 2012 at 5:58 PM, W.P. McNeill <billmcn@gmail.com> wrote:
>
>> The Hadoop framework reuses Writable objects for key and value arguments,
>> so if your code stores a pointer to that object instead of copying it you
>> can find yourself with mysterious duplicate objects.  This has tripped me
>> up a number of times. Details on what exactly I encountered and how I
>> fixed
>> it are here
>>
>> http://cornercases.wordpress.com/2011/03/14/serializing-complex-mapreduce-keys/
>> and
>> here
>>
>> http://cornercases.wordpress.com/2011/08/18/hadoop-object-reuse-pitfall-all-my-reducer-values-are-the-same/
>>  .
>>
>
>

--0016e6de16b1dd768904b634b021--