hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Watt <sw...@us.ibm.com>
Subject Hadoop Compression - Current Status
Date Mon, 12 Jul 2010 18:28:21 GMT
Please let me know if any of assertions are incorrect. I'm going to be 
adding any feedback to the Hadoop Wiki. It seems well documented that the 
LZO Codec is the most performant codec (
http://blog.oskarsson.nu/2009/03/hadoop-feat-lzo-save-disk-space-and.html) 
but it is GPL infected and thus it is separately maintained here - 
http://github.com/kevinweil/hadoop-lzo. 

With regards to performance, and if you are not using sequential files, 
Gzip is the next best codec to use, followed by bzip2. Hadoop has 
supported being able to process bzip2 and gzip input formats for awhile 
now but it could never split the files. i.e. it assigned one mapper per 
file. There are now 2 new features :
- Splitting bzip2 files available in 0.21.0 - 
https://issues.apache.org/jira/browse/HADOOP-4012
- Splitting gzip files (in progress but patch available) - 
https://issues.apache.org/jira/browse/MAPREDUCE-491

1) It appears most folks are using LZO. Given that it is GPL, are you not 
worried about it virally infecting your project ?
2) Is anyone using the new bzip2 or gzip file split compatible readers? 
How do you like them? General feedback?

Kind regards
Steve Watt
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message