Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of gordoslocos@gmail.com
 designates 74.125.82.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=TZqBJEuxEX9WjzwvrYKEAzxP9UaemfNHBaCGzXGQHwWR98/9Ts8ilgYQZ2HP3DZ0Sd
         frIP4xh4KE5uN12driVK7h/gN9+05GW3a3q6PRdcj0Vm5MYBceNOnwK3lSOwVtfXZ4hM
         zCNcaZsWayaUx1jGLAr/EmKS6JGhcCsEr/3QU=
MIME-Version: 1.0
In-Reply-To: <BANLkTikibmjg6vMgkdy+WKRU0utjFcjNwA@mail.gmail.com>
References: <BANLkTi=qEpjS0CeL-7SZ_fGhQ8qE7BfC=Q@mail.gmail.com>
	<BANLkTikibmjg6vMgkdy+WKRU0utjFcjNwA@mail.gmail.com>
Date: Thu, 26 May 2011 10:21:51 -0300
Message-ID: <BANLkTikYJBt13xOY4TokUio6WK534M4NHg@mail.gmail.com>
Subject: Re: Comparing
From: "Juan P." <gordoslocos@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=000e0ce0280e3c2ea304a42db459

--000e0ce0280e3c2ea304a42db459
Content-Type: text/plain; charset=ISO-8859-1

Harsh,
Thanks for your response, it was very helpful.
There are still a couple of things which are not really clear to me though.
You say that "Keys have got to be compared by the MR framework". But I'm
still not 100% sure why keys are sorted. I thought what hadoop did was,
during shuffling it chose which keys went to which reducer and then for each
key/value it checked the key and sent them to the correct node. If that was
the case then a good equals implementation could be enough. So why instead
of just *shuffling* does the MP framework *sort* the items?

Also, you were very clear about the use of RawComparator, thank you. Do you
know how RawComparable works though?

Again, thanks for your help!
Cheers,
Pony

On Thu, May 26, 2011 at 1:58 AM, Harsh J <harsh@cloudera.com> wrote:

> Pony,
>
> Keys have got to be compared by the MR framework somehow, and the way
> it does when you use Writables is by ensuring that your Key is of a
> Writable + Comparable type (WritableComparable).
>
> If you specify a specific comparator class, then that will be used;
> else the default WritableComparator will get asked if it can supply a
> comparator for use with your key type.
>
> AFAIK, the default WritableComparator wraps around RawComparator and
> does indeed deserialize the writables before applying the compare
> operation. The RawComparator's primary idea is to give you a pair of
> raw byte sequences to compare directly. Certain other serialization
> libraries (Apache Avro is one) provide ways to compare using bytes
> itself (Across different types), which can end up being faster when
> used in jobs.
>
> Hope this clears up your confusion.
>
> On Tue, May 24, 2011 at 2:06 AM, Juan P. <gordoslocos@gmail.com> wrote:
> > Hi guys,
> > I wanted to get your help with a couple of questions which came up while
> > looking at the Hadoop Comparator/Comparable architecture.
> >
> > As I see it before each reducer operates on each key, a sorting algorithm
> is
> > applied to them. *Why does Hadoop need to do that?*
> >
> > If I implement my own class and I intend to use it as a Key I must allow
> for
> > instances of my class to be compared. So I have 2 choices: I can
> implement
> > WritableComparable or I can register a WritableComparator for my
> > class. Should I fail to do either, would the Job fail?
> > If I register my WritableComparator which does not use the Comparable
> > interface at all, does my Key need to implement WritableComparable?
> > If I don't implement my Comparator and my Key implements
> WritableComparable,
> > does it mean that Hadoop will deserialize my Keys twice? (once for
> sorting,
> > and once for reducing)
> > What is RawComparable used for?
> >
> > Thanks for your help!
> > Pony
> >
>
>
>
> --
> Harsh J
>

--000e0ce0280e3c2ea304a42db459--