hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zheng Shao <zs...@facebook.com>
Subject RE: DefaultCodec vs. LZO
Date Fri, 23 Oct 2009 20:37:17 GMT
Thanks Yongqiang.

My last message was wrong - we had mapred.compress.map.output=false.
Will try to get LZO on our cluster and see.

Zheng
From: He Yongqiang [mailto:heyongqiang@software.ict.ac.cn]
Sent: Friday, October 23, 2009 1:28 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: DefaultCodec vs. LZO

I did a search, and find the below two links, hope they are helpful.
http://blog.oskarsson.nu/2009/03/hadoop-feat-lzo-save-disk-space-and.html
http://mail.jabber.org/pipermail/standards/2005-October/008768.html

I think Hadoop’s DefaultCodec is using native Zlib when possible.

Thanks,
Yongqiang
On 09-10-23 下午12:43, "Zheng Shao" <zshao@facebook.com> wrote:
We are using Hadoop 0.20 and mapred.map.output.compression.Codec is set to DefaultCodec.
We tried LZO but the performance seems very similar to DefaultCodec.

I heard of a lot of good words about LZO. So did anybody compare LZO with DefaultCodec?
Is there a big difference?

We are running CentOS release 5.2 (Final).

Thanks,
Zheng


Mime
View raw message