hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sigurd Spieckermann <sigurd.spieckerm...@gmail.com>
Subject Data locality of map-side join
Date Mon, 22 Oct 2012 20:29:15 GMT
Hi guys,

I've been trying to figure out whether a map-side join using the 
join-package does anything clever regarding data locality with respect 
to at least one of the partitions to join. To be more specific, if I 
want to join two datasets and some partition of dataset A is larger than 
the corresponding partition of dataset B, does Hadoop account for this 
and try to ensure that the map task is executed on the datanode storing 
the bigger partition thus reducing data transfer (if the other partition 
does not happen to be located on that same datanode)? I couldn't 
conclude the one or the other behavior from the source code and I 
couldn't find any documentation about this detail.

Thanks for clarifying!

View raw message