hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Piyush Kansal <piyush.kan...@gmail.com>
Subject v0.20.203: How to compress files in Reducer
Date Fri, 13 Apr 2012 00:44:42 GMT

I am creating o/p files in reducer using my own file name convention. So,
using FileSystem APIs I am dumping data in the files. I now want to
compress these files while writing so as to write lesser amount of data and
also to save the space on HDFS.

So, I tried following options, but none of them worked:
- setting the "mapred.output.compress" to true
- job.setOutputFormatClass( TextOutputFormat.class);
  TextOutputFormat.setCompressOutput(job, true);
  TextOutputFormat.setOutputCompressorClass(job, GzipCodec.class);
- I also tried looking into the exiting FileSystem and FileUtil APIs but
none of them has an API to write the file in compressed format

Can you please suggest how can I achieve the required goal.

Piyush Kansal

View raw message