hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi" <gr...@yahoo-inc.com>
Subject Re: large intermediate outputs
Date Mon, 03 Jan 2011 12:36:59 GMT
The following 2 could solve the issue to some extent. But these 2 are not
automatically done by hadoop. User needs to set things before submitting the
job.
(1) Enabling map output compression using configuration property
mapreduce.map.output.compress.
(2) Use combiner so that possibly less amount of intermediate data will be
emitted by mapper.

-Ravi

On 1/3/11 5:58 PM, "Debbie Fu" <fuyulin365@gmail.com> wrote:

> Hi,
> Is there any possibility that the intermediate output might be too large to
> store it in the local disk?
> If there is, what does hadoop do to solve the problem?
> Thanks.


Mime
View raw message