hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joey Echeverria <j...@cloudera.com>
Subject Re: WritableComparable
Date Mon, 15 Aug 2011 01:33:17 GMT
Does your compareTo() method test object pointer equality? If so, you could
be getting burned by Hadoop reusing Writable objects.

-Joey
On Aug 14, 2011 9:20 PM, "Stan Rosenberg" <srosenberg@proclivitysystems.com>
wrote:
> Hi Folks,
>
> After much poking around I am still unable to determine why I am seeing
> 'reduce' being called twice with the "same" key.
> Recall from my previous email that "sameness" is determined by 'compareTo'
> of my custom key type.
>
> AFAIK, the default WritableComparator invokes 'compareTo' for any two keys
> which are being ordered during sorting and merging.
> Is it somehow possible that a bitwise comparator is used for the spilled
map
> output rather than the default WritableComparator?
>
> I am out of clues, short of studying the "shuffling" code. If anyone can
> suggest some further debugging steps, don't be shy. :)
>
> Thanks!!!
>
> stan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message