hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Does mapred.local.dir is important factor in reducer side?
Date Mon, 31 Dec 2012 19:25:35 GMT
A simple fix, aside of applying a possible combiner, would be to raise
the number of reducers by a small margin such that per-partition data
falls in size.

On Mon, Dec 31, 2012 at 9:44 PM, Majid Azimi <majid.merkava@gmail.com> wrote:
> hi guys,
> Hadoop the definitive guide says:
> intermediate results on the mapper side is written to local disk at
> mapred.local.dir location so if this location does not have enough space the
> map will fail.
> I want to know if this is true on the reducer side. Output of all mappers
> will merge at reducer side. In which location this merge happens? If that
> location does not have enough space does reducer fail? What is the solution
> for MapReduce jobs if intermediat results for some keys is more than local
> disk of reducer?

Harsh J

View raw message