hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Susheel Kumar Gadalay <skgada...@gmail.com>
Subject Re: Question about shuffle/merge/sort phrase
Date Mon, 22 Dec 2014 04:41:29 GMT
It is the mapper which will push the o/p to the respective reducer as
soon as it completes.

The no of reducers are known at the beginning itself.
The mapper as it process the input split, generate the o/p of for each
reducer (if the mapper o/p key is eligible for the reducer).
The reducer will wait till the completion of all map tasks to start it

On 12/22/14, bit1129@163.com <bit1129@163.com> wrote:
> Could some one help me on this question? thanks.
> bit1129@163.com
> 发件人: Todd
> 发送时间: 2014-12-21 21:59
> 收件人: user@hadoop.apache.org
> 主题: Question about shuffle/merge/sort phrase
> Hi, Hadoopers,
> I got a question about shuffle/sort/merge phrase related..
> My understanding is that shuffle is used to transfer the mapper
> output(key/value pairs) from mapper node to reducer node, and merge phrase
> is used to merge all the mapper output from all mapper nodes, and sort
> phrase is used to sort the key/value pair by key,
> Then my question, whose responsibility is it that brings each key with all
> its values together (The reducer's input is a key and an iterative values).
> Thanks.

View raw message