Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of skgadalay@gmail.com designates
 209.85.212.181 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <2014122213222405278413@163.com>
References: <43cab251.4652.14a6d24f932.Coremail.bit1129@163.com>
	<201412221148321728172@163.com>
	<CACc2-Ys8KWi5=4nR11rHujS26g=KygjNsakrs+o+sZxZhz5eqw@mail.gmail.com>
	<CACc2-Yv9K-djH9aiThyuUjxtrD3bKPsC-4mmWnrNkphOs1g0Aw@mail.gmail.com>
	<2014122213222405278413@163.com>
Date: Mon, 22 Dec 2014 11:10:38 +0530
Message-ID: 
 <CACc2-YtyxNt3QJQJXh2vz7ULk2OLfBG3VO3JKBN=DQwk4UtFJw@mail.gmail.com>
Subject: Re: Re: Question about shuffle/merge/sort phrase
From: Susheel Kumar Gadalay <skgadalay@gmail.com>
To: user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

What I explained is shuffle phase.

After the reducer pulls the data, it does a sort on the key part only
and calls the corresponding reduce method.
On 12/22/14, bit1129@163.com <bit1129@163.com> wrote:
> Then what exactly happens after Reducer pulls all mapper output key/value
> pairs from all the mapper nodes before reducer see the
> <key,value1,value2..>?
>
>
>
> bit1129@163.com
>
> From: Susheel Kumar Gadalay
> Date: 2014-12-22 13:20
> To: user
> Subject: Re: Question about shuffle/merge/sort phrase
> Sorry, typo
>
> It is the reducer which will pull the mapper o/p as soon as it completes.
>
> On 12/22/14, Susheel Kumar Gadalay <skgadalay@gmail.com> wrote:
>> It is the mapper which will push the o/p to the respective reducer as
>> soon as it completes.
>>
>> The no of reducers are known at the beginning itself.
>> The mapper as it process the input split, generate the o/p of for each
>> reducer (if the mapper o/p key is eligible for the reducer).
>> The reducer will wait till the completion of all map tasks to start it
>> processing.
>>
>>
>> On 12/22/14, bit1129@163.com <bit1129@163.com> wrote:
>>> Could some one help me on this question? thanks.
>>>
>>>
>>>
>>> bit1129@163.com
>>>
>>> =E5=8F=91=E4=BB=B6=E4=BA=BA=EF=BC=9A Todd
>>> =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4=EF=BC=9A 2014-12-21 21:59
>>> =E6=94=B6=E4=BB=B6=E4=BA=BA=EF=BC=9A user@hadoop.apache.org
>>> =E4=B8=BB=E9=A2=98=EF=BC=9A Question about shuffle/merge/sort phrase
>>> Hi, Hadoopers,
>>> I got a question about shuffle/sort/merge phrase related..
>>> My understanding is that shuffle is used to transfer the mapper
>>> output(key/value pairs) from mapper node to reducer node, and merge
>>> phrase
>>> is used to merge all the mapper output from all mapper nodes, and sort
>>> phrase is used to sort the key/value pair by key,
>>> Then my question, whose responsibility is it that brings each key with
>>> all
>>> its values together (The reducer's input is a key and an iterative
>>> values).
>>>
>>>
>>> Thanks.
>>>
>>
>