spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghavendra (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-22622) OutOfMemory thrown by Closure Serializer without proper failure propagation
Date Wed, 29 Nov 2017 01:51:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-22622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269917#comment-16269917
] 

Raghavendra commented on SPARK-22622:
-------------------------------------

The output of previous Stage is around 70GB and my driver memory is 225GB. Why am i getting
an OOM when there is sufficient memory.
Also, since this is closure serializer, isnt it only serializing the Physical Plan and not
the RDD?
17/11/28 07:22:53 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD
[25]
at parquet at LogProcessor.java:473), which has no missing parents
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 25 - 0 - 0 - StorageLevel(1
replicas) - parquet at *******.java:473
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 20 - 0 - 0 - StorageLevel(1
replicas) - persist at *******.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 19 - 0 - 0 - StorageLevel(1
replicas) - persist at *******.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 12 - 0 - 0 - StorageLevel(1
replicas) - persist at *******.java:88
17/11/28 07:22:53 ERROR LogListener: RDD Info : FileScanRDD - 11 - 0 - 0 - StorageLevel(1
replicas) - persist at *******.java:88
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 22 - 0 - 0 - StorageLevel(1
replicas) - persist at *******.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 18 - 0 - 0 - StorageLevel(1
replicas) - persist at *******.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 21 - 0 - 0 - StorageLevel(1
replicas) - persist at *******.java:283
17/11/28 07:22:53 ERROR LogListener: RDD Info : MapPartitionsRDD - 24 - 0 - 0 - StorageLevel(1
replicas) - parquet at *******.java:473
Exception in thread "dag-scheduler-event-loop" java.lang.OutOfMemoryError
at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1677)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)

> OutOfMemory thrown by Closure Serializer without proper failure propagation
> ---------------------------------------------------------------------------
>
>                 Key: SPARK-22622
>                 URL: https://issues.apache.org/jira/browse/SPARK-22622
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.2.0
>         Environment: Spark 2.2.0
> Hadoop 2.9.0
>            Reporter: Raghavendra
>            Priority: Critical
>
> While moving from a Stage to another, the Closure serializer is trying to Serialize the
Closures and throwing OOMs.
>  This is happening when the RDD size crosses 70 GB. 
> I set the Driver Memory to 225 GB and yet the error persist.
>  There are two issues here
> * OOM thrown when there is almost 3 times of Driver memory provided than the last Stage
RDD size.(Even tried caching this into the disk before moving it into the current stage)
> * After the Error is thrown, the Spark Job does not exit. it just continues in the same
state without propagating the error into the Spark UI.
> *Scenario 1*
> {color:red}Exception in thread "dag-scheduler-event-loop" java.lang.OutOfMemoryError:
Requested array size exceeds VM limit
> 	at java.util.Arrays.copyOf(Arrays.java:3236)
> 	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> 	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> 	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> 	at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
> 	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> 	at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
> 	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> 	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
> 	at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> 	at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
> 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
> 	at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1677)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {color}
> *Scenario 2*
> {color:red}
>        Exception in thread "dag-scheduler-event-loop" java.lang.OutOfMemoryError
> 	at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)
> 	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:117)
> 	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> 	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> 	at org.apache.spark.util.ByteBufferOutputStream.write(ByteBufferOutputStream.scala:41)
> 	at java.io.ObjectOutputStream$BlockDataOutputStream.drain(ObjectOutputStream.java:1877)
> 	at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(ObjectOutputStream.java:1786)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1189)
> 	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> 	at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
> 	at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
> 	at org.apache.spark.scheduler.DAGScheduler.submitMissingTasks(DAGScheduler.scala:1003)
> 	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:930)
> 	at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:874)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1677)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1669)
> 	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1658)
> 	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
> {color}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message