hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandeep Khurana <skhurana...@gmail.com>
Subject Re: Re: Question about shuffle/merge/sort phrase
Date Mon, 22 Dec 2014 07:10:34 GMT
After reducer has the data then it does sorting and merging at its end
too.After merging (data of same key together), it passes the keys and the
collection of values for each key to reducer 1 by 1 as you said. The sort
and merge on reducer side do this merging of records of same key (after
sorting).

Have  a look at the diagram at
http://hadoop-gyan.blogspot.in/2012/11/map-reduce-shuffle-and-sort.html
 (its not my blog).

On Mon, Dec 22, 2014 at 10:52 AM, bit1129@163.com <bit1129@163.com> wrote:

> Then what exactly happens after Reducer pulls all mapper output key/value
> pairs from all the mapper nodes before reducer see the
> <key,value1,value2..>?
>
> ------------------------------
> bit1129@163.com
>
>
> *From:* Susheel Kumar Gadalay <skgadalay@gmail.com>
> *Date:* 2014-12-22 13:20
> *To:* user <user@hadoop.apache.org>
> *Subject:* Re: Question about shuffle/merge/sort phrase
> Sorry, typo
>
> It is the reducer which will pull the mapper o/p as soon as it completes.
>
> On 12/22/14, Susheel Kumar Gadalay <skgadalay@gmail.com> wrote:
> > It is the mapper which will push the o/p to the respective reducer as
> > soon as it completes.
> >
> > The no of reducers are known at the beginning itself.
> > The mapper as it process the input split, generate the o/p of for each
> > reducer (if the mapper o/p key is eligible for the reducer).
> > The reducer will wait till the completion of all map tasks to start it
> > processing.
> >
> >
> > On 12/22/14, bit1129@163.com <bit1129@163.com> wrote:
> >> Could some one help me on this question? thanks.
> >>
> >>
> >>
> >> bit1129@163.com
> >>
> >> 发件人: Todd
> >> 发送时间: 2014-12-21 21:59
> >> 收件人: user@hadoop.apache.org
> >> 主题: Question about shuffle/merge/sort phrase
> >> Hi, Hadoopers,
> >> I got a question about shuffle/sort/merge phrase related..
> >> My understanding is that shuffle is used to transfer the mapper
> >> output(key/value pairs) from mapper node to reducer node, and merge
> >> phrase
> >> is used to merge all the mapper output from all mapper nodes, and sort
> >> phrase is used to sort the key/value pair by key,
> >> Then my question, whose responsibility is it that brings each key with
> >> all
> >> its values together (The reducer's input is a key and an iterative
> >> values).
> >>
> >>
> >> Thanks.
> >>
> >
>
>


-- 
Thanks and regards
Sandeep Khurana

Mime
View raw message