hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi" <gr...@yahoo-inc.com>
Subject Re: large intermediate outputs
Date Mon, 03 Jan 2011 12:36:59 GMT
The following 2 could solve the issue to some extent. But these 2 are not
automatically done by hadoop. User needs to set things before submitting the
(1) Enabling map output compression using configuration property
(2) Use combiner so that possibly less amount of intermediate data will be
emitted by mapper.


On 1/3/11 5:58 PM, "Debbie Fu" <fuyulin365@gmail.com> wrote:

> Hi,
> Is there any possibility that the intermediate output might be too large to
> store it in the local disk?
> If there is, what does hadoop do to solve the problem?
> Thanks.

View raw message