hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: How to Compress output of mapper
Date Sun, 14 Nov 2010 12:07:58 GMT

On Sat, Nov 13, 2010 at 12:33 PM, Sujitha <gscse@tce.edu> wrote:
> Hi all,
> I am using Hadoop 0.20.2.I tried to compress the output of the mapper
> using setMapOutputCompressorClass(DefaultCodec.class) on the Job
> object,but it throws an error stating setMapOutputCompressorClass is not
> available for Job class.

That method does not exist for the Job API (seeing the list in:

You should set it manually using "mapred.compress.map.output" (bool)
and "mapred.map.output.compression.codec" (string) properties.

It does exist for the 0.20.2's "stable" JobConf API though, and in
0.21 its doable by
Job.getConfiguration().setBoolean(Job.MAP_OUTPUT_COMPRESS, bool) and
Job.getConfiguration().setClass(Job.MAP_OUTPUT_COMPRESS_CODEC, Class,

0.20.2 + New API really is misleading, but things should be fine in
0.20.3 if MAPREDUCE-1734 gets applied.

Harsh J

View raw message