Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: "Barry, Sean F" <sean.f.barry@intel.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Subject: RE: Shuffle/sort
Thread-Topic: Shuffle/sort
Thread-Index: Ac1DbQBTjDGM6kR2RcC27bYOG9lz2wAW9LoAAA37vAA=
Date: Wed, 6 Jun 2012 16:24:39 +0000
Message-ID: 
 <E4B77C96967B694B8E85FBFC19F3C14A1AC04F4A@ORSMSX103.amr.corp.intel.com>
References: 
 <E4B77C96967B694B8E85FBFC19F3C14A1AC03D8F@ORSMSX103.amr.corp.intel.com>
 <CAOcnVr30HdrTBtw3oN4ws7KxsLUa1wiRo+X9n68iTNGRCZkTDA@mail.gmail.com>
In-Reply-To: 
 <CAOcnVr30HdrTBtw3oN4ws7KxsLUa1wiRo+X9n68iTNGRCZkTDA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks Harsh!
And is this the right source code for the shuffling that is done in the red=
uce task?

http://search-hadoop.com/c/Hadoop:/hadoop-mapreduce-project/hadoop-mapreduc=
e-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapre=
duce/task/reduce/Shuffle.java%7C%7Cshuffle+sort

-sb

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]=20
Sent: Tuesday, June 05, 2012 7:43 PM
To: common-user@hadoop.apache.org
Subject: Re: Shuffle/sort

Hey Sean,

Check out http://www.slideshare.net/jhammerb/hadoop-map-reduce-arch-106883,
a slightly dated and MR1-oriented presentation from Owen O'Malley that goes=
 a good level in-depth to get an overview of how things work (including how=
 reduces pull data).

After that, check out Chris Douglas'
http://www.slideshare.net/hadoopusergroup/ordered-record-collection
that goes in-depth into the evolution of the implementations of that layer.=
 This is pretty much the state of 0.20/1.0 today too, and in 2.0 we have ha=
d Netty replacing Jetty among other improvements but I haven't a public doc=
ument link to share on this yet. Others may share the changes docs on 2.0 i=
f they have a link to one (or I'll respond back as soon as I have one).

I hope this helps!

On Wed, Jun 6, 2012 at 4:16 AM, Barry, Sean F <sean.f.barry@intel.com> wrot=
e:
> "I was always wondering after mapping, how each reduce task get its=20
> input. It is said in google's paper and hadoop's documentation that a=20
> sort is done to aggregate the same key of the map output. But there is=20
> no detailed explanation of how it is implemented and my intuition is=20
> that perhaps a global hashing will work better than sorting. So I=20
> really want to know the details and see whether my intuition is right. If=
 I can find out that in the source code, where should I start with?"
>
> I saw this question online and no one replied to it. does anyone know whe=
re I go to study the source code for the shuffle and sort.
>
> -sean


--
Harsh J