hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Douglas <chri...@yahoo-inc.com>
Subject Re: When does reducer read mapper's intermediate result?
Date Mon, 14 Jul 2008 23:04:13 GMT
Not quite; the intermediate output is written to the local disk on the  
node executing MapTask and fetched over HTTP by the ReduceTask. The  
ReduceTask need only wait for the MapTask to complete successfully  
before fetching its output, but it cannot start before all MapTasks  
have finished. The intermediate output is sorted, so the ReduceTask  
only needs to merge the output produced by the map and group by key  
(using the grouping comparator). -C

On Jul 14, 2008, at 3:59 PM, Mori Bellamy wrote:

> i'm pretty sure that the reducer waits for all of the map tasks'  
> output to be written to HDFS (or else i nee no use for the Combiner  
> class).  i'm not sure about your second question though. my gut  
> tells me "no"
> On Jul 14, 2008, at 3:50 PM, Kevin wrote:
>> Hi, there,
>> I am interested in the implementation details of hadoop mapred. In
>> particular, does the reducer wait till a map task ends and then fetch
>> the output (key-value pairs)? If so, is the very file produced by a
>> mapper for the reducer sorted before reducer gets it? (which means
>> that the reducer only needs to do merge sort when it gets all the
>> intermediate files from different mappers).
>> Best,
>> -Kevin

View raw message