spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-24687) When NoClassDefError thrown during task serialization will cause job hang
Date Tue, 10 Jul 2018 13:19:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-24687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16538576#comment-16538576
] 

Apache Spark commented on SPARK-24687:
--------------------------------------

User 'caneGuy' has created a pull request for this issue:
https://github.com/apache/spark/pull/21664

> When NoClassDefError thrown during task serialization will cause job hang
> -------------------------------------------------------------------------
>
>                 Key: SPARK-24687
>                 URL: https://issues.apache.org/jira/browse/SPARK-24687
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.1.0, 2.1.1
>            Reporter: zhoukang
>            Priority: Major
>         Attachments: hanging-960.png
>
>
> When below exception thrown:
> {code:java}
> Exception in thread "dag-scheduler-event-loop" java.lang.NoClassDefFoundError: Lcom/xxx/data/recommend/aggregator/queue/QueueName;
> 	at java.lang.Class.getDeclaredFields0(Native Method)
> 	at java.lang.Class.privateGetDeclaredFields(Class.java:2436)
> 	at java.lang.Class.getDeclaredField(Class.java:1946)
> 	at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)
> 	at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
> 	at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)
> 	at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)
> 	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)
> 	at java.io.ObjectOutputStream.writeClass(ObjectOutputStream.java:1212)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1119)
> 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> 	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
> 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> 	at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1377)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1173)
> 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> 	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> 	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
> {code}
> Stage will always hang.Not abort.
>  !hanging-960.png! 
> It is because NoClassDefError will no be catch up below.
> {code}
> var taskBinary: Broadcast[Array[Byte]] = null
>     try {
>       // For ShuffleMapTask, serialize and broadcast (rdd, shuffleDep).
>       // For ResultTask, serialize and broadcast (rdd, func).
>       val taskBinaryBytes: Array[Byte] = stage match {
>         case stage: ShuffleMapStage =>
>           JavaUtils.bufferToArray(
>             closureSerializer.serialize((stage.rdd, stage.shuffleDep): AnyRef))
>         case stage: ResultStage =>
>           JavaUtils.bufferToArray(closureSerializer.serialize((stage.rdd, stage.func):
AnyRef))
>       }
>       taskBinary = sc.broadcast(taskBinaryBytes)
>     } catch {
>       // In the case of a failure during serialization, abort the stage.
>       case e: NotSerializableException =>
>         abortStage(stage, "Task not serializable: " + e.toString, Some(e))
>         runningStages -= stage
>         // Abort execution
>         return
>       case NonFatal(e) =>
>         abortStage(stage, s"Task serialization failed: $e\n${Utils.exceptionString(e)}",
Some(e))
>         runningStages -= stage
>         return
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message