hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Da Zheng <zhengda1...@gmail.com>
Subject Re: Check lzo is working on intermediate data
Date Thu, 24 Feb 2011 15:33:40 GMT
I use the first one, and it seems to work because I see the size of data output
from mappers is much smaller.

Da

On 2/24/11 10:12 AM, Marc Sturlese wrote:
> 
> Hey there,
> I am using hadoop 0.20.2. I 've successfully installed LZOCompression
> following these steps:
> https://github.com/kevinweil/hadoop-lzo
> 
> I have some MR jobs written with the new API and I want to compress
> intermediate data.
> Not sure if my mapred-site.xml should have the properties:
> 
>   <property>
>     <name>mapred.compress.map.output</name>
>     <value>true</value>
>   </property>
>   <property>
>     <name>mapred.map.output.compression.codec</name>
>     <value>com.hadoop.compression.lzo.LzoCodec</value>
>   </property>
> 
> or:
> 
>   <property>
>     <name>mapreduce.map.output.compress</name>
>     <value>true</value>
>   </property>
>   <property>
>     <name>mapreduce.map.output.compress.codec</name>
>     <value>com.hadoop.compression.lzo.LzoCodec</value>
>   </property>
> 
> How can I check that the compression is been applied?
> 
> Thanks in advance
> 


Mime
View raw message