hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zizon Qiu <zzd...@gmail.com>
Subject Re: CompressionCodec in MapReduce
Date Wed, 11 Apr 2012 08:12:34 GMT
append your custom codec full class name in "io.compression.codecs" either
in mapred-site.xml or in the configuration object pass to Job constructor.

the map reduce framework will try to guess the compress algorithm using the
input files suffix.

if any CompressionCodec.getDefaultExtension() register in the configuration
match the suffix,hadoop will try to instantiate the codec and decompress
for you ,if succeed,automatically.

the default value for "io.compression.codecs" is

On Wed, Apr 11, 2012 at 3:55 PM, Grzegorz Gunia

> Hello,
> I am trying to apply a custom CompressionCodec to work with MapReduce
> jobs, but I haven't found a way to inject it during the reading of input
> data, or during the write of the job results.
> Am I missing something, or is there no support for compressed files in the
> filesystem?
> I am well aware of how to set it up to work during the intermitent phases
> of the MapReduce operation, but I just can't find a way to apply it BEFORE
> the job takes place...
> Is there any other way except simply uncompressing the files I need prior
> to scheduling a job?
> Huge thanks for any help you can give me!
> --
> Greg

View raw message