hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: Does mapred.local.dir is important factor in reducer side?
Date Mon, 31 Dec 2012 19:17:23 GMT
Hadoop, The Definitive Guide is only talking about Apache, CDH and
Hortonworks here.

The MapR distribution does not have this limitation and thus is one
solution for this problem.

Another solution is to do partial aggregates such as with a combiner.

On Mon, Dec 31, 2012 at 8:14 AM, Majid Azimi <majid.merkava@gmail.com>wrote:

> Hadoop the definitive guide says:
> intermediate results on the mapper side is written to local disk at
> mapred.local.dir location so if this location does not have enough space
> the map will fail.
> I want to know if this is true on the reducer side. Output of all mappers
> will merge at reducer side. In which location this merge happens? If that
> location does not have enough space does reducer fail? What is the solution
> for MapReduce jobs if intermediat results for some keys is more than local
> disk of reducer?

View raw message