hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satheesh Kumar <nks...@gmail.com>
Subject Re: 答复: MapReduce shuffle question
Date Fri, 03 Aug 2012 17:46:52 GMT
Thank you. One more follow up question:

Are there any optimizations to run map and reduces on the same nodes so
that data is not transported across the network? Generally, how often and
what % of map output is actually transferred over the network to reduce
nodes?

Thanks,
Satheesh

On Fri, Aug 3, 2012 at 7:33 AM, 梁李印 <liyin.liangly@aliyun-inc.com> wrote:

> When a map task is done, its output is always flushed to the disk and
> merged
> to one file.
> The benefit is that if the reducer is failed, the map need not to re-run.
>
> Liyin Liang
>
> -----邮件原件-----
> 发件人: Satheesh Kumar [mailto:nkseam@gmail.com]
> 发送时间: 2012年8月3日 21:23
> 收件人: common-user@hadoop.apache.org
> 主题: MapReduce shuffle question
>
> Team, can someone please clarify the following question?
>
> In the map phase, the map output is written to the local disk. And in the
> shuffle phase, the map output partitions are transferred to reduce nodes
> using http. So, my question is assuming there are no spills (data set is
> small enough to accommodate this), will the map output be transferred
> directly from memory to the reduce nodes using http without a disk access
> to write the map output? Or, is the map output always flushed to the disk
> before transferred to reduce nodes?
>
> Appreciate the help.
>
> Thanks,
> Satheesh
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message