hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: WritableComparable
Date Mon, 15 Aug 2011 02:25:08 GMT
What are the types of key1 and key2? What does the readFields() method
look like?

-Joey

On Sun, Aug 14, 2011 at 10:07 PM, Stan Rosenberg
<srosenberg@proclivitysystems.com> wrote:
> On Sun, Aug 14, 2011 at 9:33 PM, Joey Echeverria <joey@cloudera.com> wrote:
>
>> Does your compareTo() method test object pointer equality? If so, you could
>> be getting burned by Hadoop reusing Writable objects.
>
>
> Yes, but only the equality between enum values.  Interestingly, when
> 'reduce' is called there are three instances of the "same" key.
> Two instances are correctly merged and they both come from the same mapper.
>  The other instance comes from a different mapper, and for
> some reason does not get merged.  I see the key and the values
> (corresponding to the two merged instances) passed as arguments
> to 'reduce'; then in subsequent 'reduce' call I see the key and the value
> corresponding to the third instance.
>
> For completeness, here is my 'Key.compareTo':
>
> public int compareTo(Key o) {
>   if (this.type != o.type) {
>       // Type.X < Type.Y
>       return (this.type == Type.X ? -1 : 1);
>   }
>   // otherwise, delegate
>   if (this.type == Type.X) {
>      return this.key1.compareTo(o.key1);
>   } else {
>      return this.key2.compareTo(o.key2);
>   }
> }
>
> The 'type' field is an enum with two possible values, say X and Y.  Key is
> essentially a union type; i.e., at any given time
> it's the values in key1 or key2 that are being compared (depending on the
> 'type' value).
>



-- 
Joseph Echeverria
Cloudera, Inc.
443.305.9434

Mime
View raw message