hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robin Anil <robin.a...@gmail.com>
Subject Re: Enabling Intermediate Map Output compression
Date Fri, 23 Oct 2009 00:32:49 GMT
read the third conf parameter as mapred.map.output.compression.codec instead
of mapred.output.compress



On Fri, Oct 23, 2009 at 4:13 AM, Robin Anil <robin.anil@gmail.com> wrote:

> Hi,       I am using hadoop 0.20.0 release version. I was trying to enable
> intermediate output compression from code. (libhadoop.so.* was put in
> /usr/lib)
> I tried the following and it didnt give me the desired effect. Doing the
> same using old API seemed to work( JobConf.setCompressMapOutput(true))
>
> See the code below.
>
>     Configuration conf = new Configuration();
>
>     conf.set("mapred.compress.map.output", "true");
>     conf.set("mapred.output.compression.type", "BLOCK");
>     conf.set("mapred.output.compress",
>         "org.apache.hadoop.io.compress.GzipCodec");
>     Job job = new Job(conf, "Generating dataset based from input");
>
>
> Please help
>
> Robin
>
>

Mime
View raw message