hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Errors reading lzo-compressed files from Hadoop
Date Tue, 06 Apr 2010 17:35:58 GMT
Hi Alex,
Unfortunately I wasn't able to reproduce, and the data Dmitriy is
working with is sensitive.
Do you have some data you could upload (or send me off list) that
exhibits the issue?
-Todd

On Tue, Apr 6, 2010 at 9:50 AM, Alex Roetter <aroetter@imageshack.net> wrote:
>
> Todd Lipcon <todd@...> writes:
>
> >
> > Hey Dmitriy,
> >
> > This is very interesting (and worrisome in a way!) I'll try to take a look
> > this afternoon.
> >
> > -Todd
> >
>
> Hi Todd,
>
> I wanted to see if you made any progress on this front. I'm seeing a very
> similar error, trying to run a MR (Hadoop 0.20.1) over a bunch of
> LZOP compressed / indexed files (using Kevin Weil's package), and I have one
> map task that always fails in what looks like the same place as described in
> the previous post. I haven't yet done the experimentation mentioned above
> (isolating the input file corresponding to the failed map task, decompressing
> it / recompressing it, testing it out operating directly on local disk
> instead of HDFS, etc).
>
> However, since I am crashing in exactly the same place it seems likely this
> is related, and thought I'd check on your work in the meantime.
>
> FYI, my stack track is below:
>
> 2010-04-05 18:15:16,895 FATAL org.apache.hadoop.mapred.TaskTracker: Error
> running child : java.lang.InternalError: lzo1x_decompress_safe returned:
>        at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect
> (Native Method)
>        at com.hadoop.compression.lzo.LzoDecompressor.decompress
> (LzoDecompressor.java:303)
>        at
> com.hadoop.compression.lzo.LzopDecompressor.decompress
> (LzopDecompressor.java:104)
>        at com.hadoop.compression.lzo.LzopInputStream.decompress
> (LzopInputStream.java:223)
>        at
> org.apache.hadoop.io.compress.DecompressorStream.read
> (DecompressorStream.java:74)
>        at java.io.InputStream.read(InputStream.java:85)
>        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
>        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:187)
>        at
> com.hadoop.mapreduce.LzoLineRecordReader.nextKeyValue
> (LzoLineRecordReader.java:126)
>        at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue
> (MapTask.java:423)
>        at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(MapContext.java:67)
>        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
>        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:583)
>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>        at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> Any update much appreciated,
> Alex
>
>
>
>
>



--
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message