hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Eastman" <jeast...@collab.net>
Subject RE: Decompression Blues
Date Wed, 27 Feb 2008 16:56:37 GMT
I unzipped and rezipped all the files using gzip 1.3.3 and uploaded the
files again. I got the same exceptions.

I set the hadoop.native.lib property to false and bounced the cloud,
then ran my job. I still get the same exceptions.

Any more suggestions?

Jeff

-----Original Message-----
From: Arun C Murthy [mailto:acm@yahoo-inc.com] 
Sent: Tuesday, February 26, 2008 3:47 PM
To: core-user@hadoop.apache.org
Subject: Re: Decompression Blues

Jeff,

On Feb 26, 2008, at 12:58 PM, Jeff Eastman wrote:

> I'm processing a number of .gz compressed Apache and other logs using
> Hadoop 0.15.2 and encountering fatal decompression errors such as:
>
>

How did you compress your input files? Could you share details on the  
version of your gzip and other tools?

Try setting "hadoop.native.lib" property to 'false' via  
NativeCodeLoader.setLoadNativeLibraries for you job and see how it  
works...

Arun

>
> 08/02/26 12:09:12 INFO mapred.JobClient: Task Id :
> task_200802171116_0001_m_000005_0, Status : FAILED
>
> java.lang.InternalError
>
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.init(Native  
> Method)
>
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init> 
> (ZlibDecompres
> sor.java:111)
>
>         at
> org.apache.hadoop.io.compress.GzipCodec.createDecompressor 
> (GzipCodec.jav
> a:188)
>
>         at
> org.apache.hadoop.io.compress.GzipCodec.createInputStream 
> (GzipCodec.java
> :170)
>
>         at
> org.apache.hadoop.mapred.LineRecordReader.<init> 
> (LineRecordReader.java:7
> 5)
>
>         at
> org.apache.hadoop.mapred.TextInputFormat.getRecordReader 
> (TextInputFormat
> .java:50)
>
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:156)
>
>         at
> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1787)
>
>
>
> I looked in Jira but did not find any issues. Is this pilot error?  
> Some
> of the files work just fine. Is there a workaround besides  
> unzipping all
> the files in the DFS?
>
>
>
> Jeff
>


Mime
View raw message