Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of msegel@navteq.com designates
 204.120.70.37 as permitted sender)
From: "Segel, Mike" <msegel@navteq.com>
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
Date: Mon, 12 Jul 2010 13:42:09 -0500
Subject: RE: Hadoop Compression - Current Status
Thread-Topic: Hadoop Compression - Current Status
Thread-Index: Acsh8CJQR6TU1ksZRpOJ+9hfYCVaPQAAZ8ig
Message-ID: 
 <ABC24175AFD3BE4DA15F4CD375ED413D0608592CDE@hq-ex-mb02.ad.navteq.com>
References: <AANLkTimDEa5h1VyCcuo6BbmMvB3HwXSUlV0xqUyvmtAs@mail.gmail.com>
	<AANLkTilvOboVrQW8P89MM8tjhw0-HVYNsMFCODQ0jVh7@mail.gmail.com>
 <AANLkTikj1kYZijxJ3xCtegV0RE-u9gU-A_5MqOPNNdpF@mail.gmail.com>
 <OF3EF906E2.3F172DD0-ON8625775E.0064681A-8625775E.00657990@us.ibm.com>
In-Reply-To: 
 <OF3EF906E2.3F172DD0-ON8625775E.0064681A-8625775E.00657990@us.ibm.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

How can you say zip files are 'best codecs' to use?

Call me silly but I seem to recall that if you're using a zip'd file for =
input you can't really use a file splitter?
(Going from memory, which isn't the best thing to do...)

-Mike


-----Original Message-----
From: Stephen Watt [mailto:swatt@us.ibm.com]=20
Sent: Monday, July 12, 2010 1:28 PM
To: common-dev@hadoop.apache.org
Subject: Hadoop Compression - Current Status

Please let me know if any of assertions are incorrect. I'm going to be=20
adding any feedback to the Hadoop Wiki. It seems well documented that the=
=20
LZO Codec is the most performant codec (
http://blog.oskarsson.nu/2009/03/hadoop-feat-lzo-save-disk-space-and.html=
)=20
but it is GPL infected and thus it is separately maintained here -=20
http://github.com/kevinweil/hadoop-lzo.=20

With regards to performance, and if you are not using sequential files,=20
Gzip is the next best codec to use, followed by bzip2. Hadoop has=20
supported being able to process bzip2 and gzip input formats for awhile=20
now but it could never split the files. i.e. it assigned one mapper per=20
file. There are now 2 new features :
- Splitting bzip2 files available in 0.21.0 -=20
https://issues.apache.org/jira/browse/HADOOP-4012
- Splitting gzip files (in progress but patch available) -=20
https://issues.apache.org/jira/browse/MAPREDUCE-491

1) It appears most folks are using LZO. Given that it is GPL, are you not=
=20
worried about it virally infecting your project ?
2) Is anyone using the new bzip2 or gzip file split compatible readers?=20
How do you like them? General feedback?

Kind regards
Steve Watt


The information contained in this communication may be CONFIDENTIAL and i=
s intended only for the use of the recipient(s) named above.  If you are =
not the intended recipient, you are hereby notified that any disseminatio=
n, distribution, or copying of this communication, or any of its contents=
, is strictly prohibited.  If you have received this communication in err=
or, please notify the sender and delete/destroy the original message and =
any copy of it from your computer or paper files.