hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@hortonworks.com>
Subject Re: How reduce tasks know which partition they should read?
Date Mon, 09 Mar 2015 22:21:14 GMT

The reducers(Fetcher.java) simply ask the Shuffle Service (ShuffleHandler.java) to give them
output corresponding to a specific map. The partitioning detail is hidden from the reducers.

Thanks,
+Vinod

On Mar 9, 2015, at 7:56 AM, xeonmailinglist-gmail <xeonmailinglist@gmail.com> wrote:

> Hi,
> 
> I am looking to the Yarn mapreduce internals to try to understand how reduce tasks know
which partition of the map output they should read. Even, when they re-execute after a crash?
> 
> I am also looking to the mapreduce source code. Is there any class that I should look
to try to understand this question?
> 
> Any help?
> 
> Thanks
> 
> 
> -- 
> --
> 


Mime
View raw message