spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allan Douglas R. de Oliveira (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle
Date Sat, 09 May 2015 20:37:02 GMT

    [ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14536880#comment-14536880 ] 

Allan Douglas R. de Oliveira edited comment on SPARK-4105 at 5/9/15 8:36 PM:
-----------------------------------------------------------------------------

Got something like this but using:
- Java serializer
- Snappy
- Also Lz4
- Spark 1.3.0 (most things on default settings)

We got the error many times on the same cluster (which was doing fine for days) but after recreating it the problem disappeared again. Stack traces (the first two are from two different runs using Snappy and the third from an execution using Lz4):

15/05/09 13:05:55 ERROR Executor: Exception in task 234.2 in stage 27.1 (TID 876507)
java.io.IOException: PARSING_ERROR(2)
		at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:84)
		at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
		at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:594)
		at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:358)
		at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:387)
		at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
		at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
		at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
		at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
		at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
		at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:133)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:125)
		at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:60)
		at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:46)
		at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
		at org.apache.spark.scheduler.Task.run(Task.scala:64)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)



15/05/08 20:46:28 WARN scheduler.TaskSetManager: Lost task 9559.0 in stage 55.0 (TID 424644, ip-172-24-36-214.ec2.internal): java.io.IOException: FAILED_TO_UNCOMPRESS(5)
		at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:84)
		at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
		at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:444)
		at org.xerial.snappy.Snappy.uncompress(Snappy.java:480)
		at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:362)
		at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:387)
		at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
		at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
		at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
		at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
		at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
		at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:606)
		at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
		at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
		at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:133)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:125)
		at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:60)
		at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:46)
		at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
		at org.apache.spark.scheduler.Task.run(Task.scala:64)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
		
		
		
15/05/09 16:27:01 WARN scheduler.TaskSetManager: Lost task 1365.0 in stage 52.0 (TID 812600, ip-172-24-33-2.ec2.internal): FetchFailed(BlockManagerId(19, ip-172-24-33-64.ec2.internal, 34224), shuffleId=14, mapId=662, reduceId=1365, message=
org.apache.spark.shuffle.FetchFailedException: Stream is corrupted
		at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$.org$apache$spark$shuffle$hash$BlockStoreShuffleFetcher$$unpackBlock$1(BlockStoreShuffleFetcher.scala:67)
		at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$$anonfun$3.apply(BlockStoreShuffleFetcher.scala:83)
		at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$$anonfun$3.apply(BlockStoreShuffleFetcher.scala:83)
		at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:91)
		at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:44)
		at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
		at org.apache.spark.scheduler.Task.run(Task.scala:64)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Stream is corrupted
		at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:152)
		at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:116)
		at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
		at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
		at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
		at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
		at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
		at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63)
		at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63)
		at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:102)
		at org.apache.spark.storage.ShuffleBlockFetcherIterator$$anonfun$4.apply(ShuffleBlockFetcherIterator.scala:302)
		at org.apache.spark.storage.ShuffleBlockFetcherIterator$$anonfun$4.apply(ShuffleBlockFetcherIterator.scala:300)
		at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
		at scala.util.Try$.apply(Try.scala:161)
		at scala.util.Success.map(Try.scala:206)
		at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:300)
		at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
		... 51 more


The cluster was 15 d2.4xlarge using customized spark_ec2.py (but on stock image).

After the problem started it happened in 18 from 20 executions (i.e only 2 executions finished successfully). Curiously it happened after a deploy which updated the application jar (but this wasn't the cause because rolling back didn't fix it). My guess is that killing the running job (as consequence of the deploy) left the cluster in a inconsistent state which new executions couldn't handle.


was (Author: douglaz):
Got something like this but using:
- Java serializer
- Snappy
- Also Lz4
- Spark 1.3.0 (most things on default settings)

We got the error many times on the same cluster (which was doing fine for days) but after recreating it the problem disappeared again. Stack traces (the first two are from two different runs using Snappy and the third from an execution using Lz4):

15/05/09 13:05:55 ERROR Executor: Exception in task 234.2 in stage 27.1 (TID 876507)
java.io.IOException: PARSING_ERROR(2)
		at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:84)
		at org.xerial.snappy.SnappyNative.uncompressedLength(Native Method)
		at org.xerial.snappy.Snappy.uncompressedLength(Snappy.java:594)
		at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:358)
		at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:387)
		at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
		at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
		at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
		at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
		at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
		at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:133)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:125)
		at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:60)
		at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:46)
		at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
		at org.apache.spark.scheduler.Task.run(Task.scala:64)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)



15/05/08 20:46:28 WARN scheduler.TaskSetManager: Lost task 9559.0 in stage 55.0 (TID 424644, ip-172-24-36-214.ec2.internal): java.io.IOException: FAILED_TO_UNCOMPRESS(5)
		at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:84)
		at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
		at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:444)
		at org.xerial.snappy.Snappy.uncompress(Snappy.java:480)
		at org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:362)
		at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:387)
		at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2293)
		at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2586)
		at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318)
		at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
		at scala.collection.immutable.$colon$colon.readObject(List.scala:366)
		at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:606)
		at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
		at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
		at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
		at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
		at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
		at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
		at org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:133)
		at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:125)
		at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:60)
		at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:46)
		at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
		at org.apache.spark.scheduler.Task.run(Task.scala:64)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
		
		
		
15/05/09 16:27:01 WARN scheduler.TaskSetManager: Lost task 1365.0 in stage 52.0 (TID 812600, ip-172-24-33-2.ec2.internal): FetchFailed(BlockManagerId(19, ip-172-24-33-64.ec2.internal, 34224), shuffleId=14, mapId=662, reduceId=1365, message=
org.apache.spark.shuffle.FetchFailedException: Stream is corrupted
		at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$.org$apache$spark$shuffle$hash$BlockStoreShuffleFetcher$$unpackBlock$1(BlockStoreShuffleFetcher.scala:67)
		at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$$anonfun$3.apply(BlockStoreShuffleFetcher.scala:83)
		at org.apache.spark.shuffle.hash.BlockStoreShuffleFetcher$$anonfun$3.apply(BlockStoreShuffleFetcher.scala:83)
		at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
		at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
		at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
		at org.apache.spark.Aggregator.combineCombinersByKey(Aggregator.scala:91)
		at org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:44)
		at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:130)
		at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:127)
		at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
		at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
		at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
		at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
		at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:127)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
		at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
		at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
		at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
		at org.apache.spark.scheduler.Task.run(Task.scala:64)
		at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
		at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Stream is corrupted
		at net.jpountz.lz4.LZ4BlockInputStream.refill(LZ4BlockInputStream.java:152)
		at net.jpountz.lz4.LZ4BlockInputStream.read(LZ4BlockInputStream.java:116)
		at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
		at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2323)
		at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2794)
		at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801)
		at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
		at org.apache.spark.serializer.JavaDeserializationStream$$anon$1.<init>(JavaSerializer.scala:63)
		at org.apache.spark.serializer.JavaDeserializationStream.<init>(JavaSerializer.scala:63)
		at org.apache.spark.serializer.JavaSerializerInstance.deserializeStream(JavaSerializer.scala:102)
		at org.apache.spark.storage.ShuffleBlockFetcherIterator$$anonfun$4.apply(ShuffleBlockFetcherIterator.scala:302)
		at org.apache.spark.storage.ShuffleBlockFetcherIterator$$anonfun$4.apply(ShuffleBlockFetcherIterator.scala:300)
		at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
		at scala.util.Try$.apply(Try.scala:161)
		at scala.util.Success.map(Try.scala:206)
		at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:300)
		at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:53)
		... 51 more


The cluster was 15 d2.4xlarge using built using customized spark_ec2.py (but on stock image)

After the problem started it happened in 18 from 20 executions (i.e only 2 executions finished successfully). Curiously it happened after a deploy which updated the application jar (but this wasn't the cause because rolling back didn't fix it). My guess is that killing the running job (as consequence of the deploy) left the cluster in a inconsistent state which new executions couldn't handle.

> FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle
> ---------------------------------------------------------------------------------
>
>                 Key: SPARK-4105
>                 URL: https://issues.apache.org/jira/browse/SPARK-4105
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle, Spark Core
>    Affects Versions: 1.2.0, 1.2.1, 1.3.0
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>            Priority: Blocker
>         Attachments: JavaObjectToSerialize.java, SparkFailedToUncompressGenerator.scala
>
>
> We have seen non-deterministic {{FAILED_TO_UNCOMPRESS(5)}} errors during shuffle read.  Here's a sample stacktrace from an executor:
> {code}
> 14/10/23 18:34:11 ERROR Executor: Exception in task 1747.3 in stage 11.0 (TID 33053)
> java.io.IOException: FAILED_TO_UNCOMPRESS(5)
> 	at org.xerial.snappy.SnappyNative.throw_error(SnappyNative.java:78)
> 	at org.xerial.snappy.SnappyNative.rawUncompress(Native Method)
> 	at org.xerial.snappy.Snappy.rawUncompress(Snappy.java:391)
> 	at org.xerial.snappy.Snappy.uncompress(Snappy.java:427)
> 	at org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:127)
> 	at org.xerial.snappy.SnappyInputStream.readHeader(SnappyInputStream.java:88)
> 	at org.xerial.snappy.SnappyInputStream.<init>(SnappyInputStream.java:58)
> 	at org.apache.spark.io.SnappyCompressionCodec.compressedInputStream(CompressionCodec.scala:128)
> 	at org.apache.spark.storage.BlockManager.wrapForCompression(BlockManager.scala:1090)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator$$anon$1$$anonfun$onBlockFetchSuccess$1.apply(ShuffleBlockFetcherIterator.scala:116)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator$$anon$1$$anonfun$onBlockFetchSuccess$1.apply(ShuffleBlockFetcherIterator.scala:115)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:243)
> 	at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:52)
> 	at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
> 	at org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:30)
> 	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:327)
> 	at org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:129)
> 	at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$5.apply(CoGroupedRDD.scala:159)
> 	at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$5.apply(CoGroupedRDD.scala:158)
> 	at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> 	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> 	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> 	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> 	at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:158)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> 	at org.apache.spark.rdd.MappedValuesRDD.compute(MappedValuesRDD.scala:31)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> 	at org.apache.spark.rdd.FlatMappedValuesRDD.compute(FlatMappedValuesRDD.scala:31)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> 	at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
> 	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> 	at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
> 	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> 	at org.apache.spark.scheduler.Task.run(Task.scala:56)
> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:181)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}
> Here's another occurrence of a similar error:
> {code}
> java.io.IOException: failed to read chunk
>         org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:348)
>         org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:159)
>         org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
>         java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2310)
>         java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2712)
>         java.io.ObjectInputStream$BlockDataInputStream.readFully(ObjectInputStream.java:2742)
>         java.io.ObjectInputStream.readArray(ObjectInputStream.java:1687)
>         java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344)
>         java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>         org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>         org.apache.spark.serializer.DeserializationStream$$anon$1.getNext(Serializer.scala:133)
>         org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:71)
>         scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
>         org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:30)
>         org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
>         org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:129)
>         org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:58)
>         org.apache.spark.shuffle.hash.HashShuffleReader.read(HashShuffleReader.scala:46)
>         org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:92)
>         org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>         org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>         org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
>         org.apache.spark.scheduler.Task.run(Task.scala:56)
>         org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:182)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
> {code}
> The first stacktrace was reported by a Spark user.  The second stacktrace occurred when running
> {code}
> import java.util.Random
> val numKeyValPairs=1000
> val numberOfMappers=200
> val keySize=10000
> for (i <- 0 to 19) {
> val pairs1 = sc.parallelize(0 to numberOfMappers, numberOfMappers).flatMap(p=>{
>   val randGen = new Random
>   val arr1 = new Array[(Int, Array[Byte])](numKeyValPairs)
>   for (i <- 0 until numKeyValPairs){
>     val byteArr = new Array[Byte](keySize)
>     randGen.nextBytes(byteArr)
>     arr1(i) = (randGen.nextInt(Int.MaxValue),byteArr)
>   }
>   arr1
> })
>   pairs1.groupByKey(numberOfMappers).count
> }
> {code}
> This job frequently runs without any problems, but when it fails it seem that every post-shuffle task fails with either PARSING_ERROR(2), FAILED_TO_UNCOMPRESS(5), or some other decompression error.  I've seen reports of similar problems when using LZF compression, so I think that this is caused by some sort of general stream corruption issue. 
> This issue has been observed even when no spilling occurs, so I don't believe that this is due to a bug in spilling code.
> I was unable to reproduce this when running this code in a fresh Spark EC2 cluster and we've been having a hard time finding a deterministic reproduction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message