hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: large intermediate outputs
Date Mon, 03 Jan 2011 12:57:32 GMT
Additionally, you can set mapred.local.dir to be a comma-separated
list of paths that reside on multiple disks -- this spreads I/O plus
gives you additional space.

But I suppose if a single Mapper is writing a huge amount of data for
a single partition output, it may cause a disk fill-up. Please correct
me if am wrong here.

On Mon, Jan 3, 2011 at 5:58 PM, Debbie Fu <fuyulin365@gmail.com> wrote:
> Hi,
> Is there any possibility that the intermediate output might be too large to
> store it in the local disk?
> If there is, what does hadoop do to solve the problem?
> Thanks.
> --
> Best regards!

Harsh J

View raw message