hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White" <...@cloudera.com>
Subject Re: Problem with Hadoop and concatenated gzip files
Date Mon, 12 Jan 2009 13:19:46 GMT
I've opened https://issues.apache.org/jira/browse/HADOOP-5014 for this.

Do you get this behaviour when you use the native libraries?

Tom

On Sat, Jan 10, 2009 at 12:26 AM, Oscar Gothberg
<Oscar.Gothberg@platform-a.com> wrote:
> Hi,
>
> I'm having trouble with Hadoop (tested with 0.17 and 0.19) not fully processing certain
gzipped input files. Basically it only actually reads and processes a first part of the gzipped
file, and just ignores the rest without any kind of warning.
>
> It affects at least (but is maybe not limited to?) any gzip files that are a result of
concatenation (which should be legal to do with gzip format):
> http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage
>
> Repro case, using the "WordCount" example from the hadoop distribution:
> $ echo 'one two three' > f1
> $ echo 'four five six' > f2
> $ gzip -c f1 > combined_file.gz
> $ gzip -c f2 >> combined_file.gz
>
> Now, if I run "WordCount" with combined_file.gz as input, it will only find the words
'one', 'two', 'three', but not 'four', 'five', 'six'.
>
> It seems Java's GZIPInputStream may have a similar issue:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4691425
>
> Now, if I unzip and re-gzip this 'combined_file.gz' manually, the problem goes away.
>
> It's especially dangerous since Hadoop doesn't show any errors or complains in the least.
It just ignores this extra input. The only way of noticing is to run one's app with gzipped-
and unzipped data side by side and notice the record counts being different.
>
> Is anyone else familiar with this problem? Any solutions, workarounds, short of re-gzipping
very large amounts of data?
>
> Thanks!
> / Oscar
>
> ________________________________
> The information transmitted in this email is intended only for the person(s) or entity
to which it is addressed and may contain confidential and/or privileged material. Any review,
retransmission, dissemination or other use of, or taking of any action in reliance upon, this
information by persons or entities other than the intended recipient is prohibited. If you
received this email in error, please contact the sender and permanently delete the email from
any computer.
>

Mime
View raw message