hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Does mapred.local.dir is important factor in reducer side?
Date Mon, 31 Dec 2012 19:28:02 GMT

Do note that the local directory configs accept URIs in 2.x releases,
allowing users to plug alternative filesystems if they wanted to.

On Tue, Jan 1, 2013 at 12:47 AM, Ted Dunning <tdunning@maprtech.com> wrote:
> Hadoop, The Definitive Guide is only talking about Apache, CDH and
> Hortonworks here.
> The MapR distribution does not have this limitation and thus is one solution
> for this problem.
> Another solution is to do partial aggregates such as with a combiner.
> On Mon, Dec 31, 2012 at 8:14 AM, Majid Azimi <majid.merkava@gmail.com>
> wrote:
>> Hadoop the definitive guide says:
>> intermediate results on the mapper side is written to local disk at
>> mapred.local.dir location so if this location does not have enough space the
>> map will fail.
>> I want to know if this is true on the reducer side. Output of all mappers
>> will merge at reducer side. In which location this merge happens? If that
>> location does not have enough space does reducer fail? What is the solution
>> for MapReduce jobs if intermediat results for some keys is more than local
>> disk of reducer?

Harsh J

View raw message