hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Riccardo Boscolo" <drbosc...@netseer.com>
Subject Re: Decompression Blues
Date Tue, 26 Feb 2008 23:03:43 GMT
Jeff,

your Hadoop native libraries appear to be loading correctly, but the libz
you have installed on your system cannot decode the header of the .gz file.
This is the line of code causing it:

stream = init(this.header.windowBits());

and the init() function is native.

RB

On 2/26/08, Jeff Eastman <jeastman@collab.net> wrote:
>
> I'm processing a number of .gz compressed Apache and other logs using
> Hadoop 0.15.2 and encountering fatal decompression errors such as:
>
>
>
> 08/02/26 12:09:12 INFO mapred.JobClient: Task Id :
> task_200802171116_0001_m_000005_0, Status : FAILED
>
> java.lang.InternalError
>
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.init(Native Method)
>
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompres
> sor.java:111)
>
>         at
> org.apache.hadoop.io.compress.GzipCodec.createDecompressor(GzipCodec.jav
> a:188)
>
>         at
> org.apache.hadoop.io.compress.GzipCodec.createInputStream(GzipCodec.java
> :170)
>
>         at
> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:7
> 5)
>
>         at
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat
> .java:50)
>
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:156)
>
>         at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
>
>
>
> I looked in Jira but did not find any issues. Is this pilot error? Some
> of the files work just fine. Is there a workaround besides unzipping all
> the files in the DFS?
>
>
>
>
> Jeff
>
>


-- 
-------------------------------
Riccardo Boscolo, PhD
V.P. of Core Technology
Netseer Inc.
11943 Montana Ave, Suite 200
Los Angeles, CA 90049
T: 310-597-4482
F: 310-597-4489
Email: drboscolo@netseer.com
-------------------------------

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message