hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: Doubt from the book "Definitive Guide"
Date Thu, 05 Apr 2012 17:57:21 GMT
On Thu, Apr 5, 2012 at 7:03 AM, Mohit Anchlia <mohitanchlia@gmail.com> wrote:
> Only advantage I was thinking of was that in some cases reducers might be
> able to take advantage of data locality and avoid multiple HTTP calls, no?
> Data is anyways written, so last merged file could go on HDFS instead of
> local disk.
> I am new to hadoop so just asking question to understand the rational
> behind using local disk for final output.

So basically it's a tradeoff here, you get more replicas to copy from
but you have 2 more copies to write. Considering that that data's very
short lived and that it doesn't need to be replicated (since if the
machine fails the maps are replayed anyway) it seems that writing 2
replicas that are potentially unused would be hurtful.

Regarding locality, it might make sense on a small cluster but the
more you add nodes the smaller the chance to have local replicas for
each blocks of data you're looking for.


View raw message