Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <ac79ea400901120519wa5ff60fq1740f53eb6353035@mail.gmail.com>
Date: Mon, 12 Jan 2009 13:19:46 +0000
From: "Tom White" <tom@cloudera.com>
To: core-user@hadoop.apache.org
Subject: Re: Problem with Hadoop and concatenated gzip files
In-Reply-To: 
 <5A1055F3C9314142A922A9FC6AC21A6106DF8CC32C@BALT-EMAIL.corp.advertising.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: 
 <5A1055F3C9314142A922A9FC6AC21A6106DF8CC32C@BALT-EMAIL.corp.advertising.com>

I've opened https://issues.apache.org/jira/browse/HADOOP-5014 for this.

Do you get this behaviour when you use the native libraries?

Tom

On Sat, Jan 10, 2009 at 12:26 AM, Oscar Gothberg
<Oscar.Gothberg@platform-a.com> wrote:
> Hi,
>
> I'm having trouble with Hadoop (tested with 0.17 and 0.19) not fully processing certain gzipped input files. Basically it only actually reads and processes a first part of the gzipped file, and just ignores the rest without any kind of warning.
>
> It affects at least (but is maybe not limited to?) any gzip files that are a result of concatenation (which should be legal to do with gzip format):
> http://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage
>
> Repro case, using the "WordCount" example from the hadoop distribution:
> $ echo 'one two three' > f1
> $ echo 'four five six' > f2
> $ gzip -c f1 > combined_file.gz
> $ gzip -c f2 >> combined_file.gz
>
> Now, if I run "WordCount" with combined_file.gz as input, it will only find the words 'one', 'two', 'three', but not 'four', 'five', 'six'.
>
> It seems Java's GZIPInputStream may have a similar issue:
> http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4691425
>
> Now, if I unzip and re-gzip this 'combined_file.gz' manually, the problem goes away.
>
> It's especially dangerous since Hadoop doesn't show any errors or complains in the least. It just ignores this extra input. The only way of noticing is to run one's app with gzipped- and unzipped data side by side and notice the record counts being different.
>
> Is anyone else familiar with this problem? Any solutions, workarounds, short of re-gzipping very large amounts of data?
>
> Thanks!
> / Oscar
>
> ________________________________
> The information transmitted in this email is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and permanently delete the email from any computer.
>