hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marc Sturlese <marc.sturl...@gmail.com>
Subject LZO exception decompressing (returned -8)
Date Tue, 28 Feb 2012 10:58:42 GMT
Hey there,
I've been running a cluster for over a year and was getting a lzo
decompressing exception less than once a month. Suddenly it happens almost
once per day. Any ideas what could be causing it? I'm with hadoop 0.20.2
I've thought in moving to snappy but would like to know why this happens
more often now

The exception happens always when the reducer gets data from the map and
looks like:

Error: java.lang.InternalError: lzo1x_decompress returned: -8
	at com.hadoop.compression.lzo.LzoDecompressor.decompressBytesDirect(Native
Method)
	at
com.hadoop.compression.lzo.LzoDecompressor.decompress(LzoDecompressor.java:305)
	at
org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:76)
	at
org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:75)
	at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1553)
	at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutput(ReduceTask.java:1432)
	at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:1285)
	at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1216)

Thanks in advance.

--
View this message in context: http://lucene.472066.n3.nabble.com/LZO-exception-decompressing-returned-8-tp3783652p3783652.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Mime
View raw message