hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Satheesh Kumar <nks...@gmail.com>
Subject MapReduce shuffle question
Date Fri, 03 Aug 2012 13:23:24 GMT
Team, can someone please clarify the following question?

In the map phase, the map output is written to the local disk. And in the
shuffle phase, the map output partitions are transferred to reduce nodes
using http. So, my question is assuming there are no spills (data set is
small enough to accommodate this), will the map output be transferred
directly from memory to the reduce nodes using http without a disk access
to write the map output? Or, is the map output always flushed to the disk
before transferred to reduce nodes?

Appreciate the help.

Thanks,
Satheesh

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message