hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koji Noguchi <knogu...@yahoo-inc.com>
Subject Re: how set compression in the map output?
Date Wed, 02 Feb 2011 15:18:14 GMT
Try
org.apache.hadoop.io.compress.GzipCodec

Koji


On 2/2/11 5:50 AM, "Pedro Costa" <psdc1978@gmail.com> wrote:

Hi,

I'm running the wordcount example, but I would like compress the map output.

I set the following properties in the mapred-site.xml

[code]
<property>
        <name>mapred.compress.map.output</name>
        <value>true</value>
</property>

<property>
        <name>mapred.map.output.compression.codec</name>
        <value>gzip</value>
</property>
[/code]

but I still got the error:

java.lang.IllegalArgumentException: Compression codec gzip was not found.
        at org.apache.hadoop.mapred.JobConf.getMapOutputCompressorClass(JobConf.java:512)
        at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:846)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:531)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:627)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
        at org.apache.hadoop.mapred.Child.main(Child.java:190)



I run the wordcount with the command:

[code]
bin/hadoop jar build/hadoop-0.20.2-dev-examples.jar wordcount
gutenberg gutenberg-output
[/code]

How I set the compression in the map output?

Thanks,
--
Pedro


Mime
View raw message