hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robin Anil <robin.a...@gmail.com>
Subject Enabling Intermediate Map Output compression
Date Thu, 22 Oct 2009 22:43:11 GMT
Hi,       I am using hadoop 0.20.0 release version. I was trying to enable
intermediate output compression from code. (libhadoop.so.* was put in
/usr/lib)
I tried the following and it didnt give me the desired effect. Doing the
same using old API seemed to work( JobConf.setCompressMapOutput(true))

See the code below.

    Configuration conf = new Configuration();

    conf.set("mapred.compress.map.output", "true");
    conf.set("mapred.output.compression.type", "BLOCK");
    conf.set("mapred.output.compress",
        "org.apache.hadoop.io.compress.GzipCodec");
    Job job = new Job(conf, "Generating dataset based from input");


Please help

Robin

Mime
View raw message