spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a...@apache.org
Subject git commit: [SPARK-2403] Catch all errors during serialization in DAGScheduler
Date Tue, 08 Jul 2014 17:44:09 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.0 4bf8ddaee -> 3bd32f023


[SPARK-2403] Catch all errors during serialization in DAGScheduler

https://issues.apache.org/jira/browse/SPARK-2403

Spark hangs for us whenever we forget to register a class with Kryo. This should be a simple
fix for that. But let me know if you have a better suggestion.

I did not write a new test for this. It would be pretty complicated and I'm not sure it's
worthwhile for such a simple change. Let me know if you disagree.

Author: Daniel Darabos <darabos.daniel@gmail.com>

Closes #1329 from darabos/spark-2403 and squashes the following commits:

3aceaad [Daniel Darabos] Print full stack trace for miscellaneous exceptions during serialization.
52c22ba [Daniel Darabos] Only catch NonFatal exceptions.
361e962 [Daniel Darabos] Catch all errors during serialization in DAGScheduler.

(cherry picked from commit c8a2313cdf825e0191680a423d17619b5504ff89)
Signed-off-by: Aaron Davidson <aaron@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3bd32f02
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3bd32f02
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3bd32f02

Branch: refs/heads/branch-1.0
Commit: 3bd32f023d9bd83da7afab37fffe614064df3e6b
Parents: 4bf8dda
Author: Daniel Darabos <darabos.daniel@gmail.com>
Authored: Tue Jul 8 10:43:46 2014 -0700
Committer: Aaron Davidson <aaron@databricks.com>
Committed: Tue Jul 8 10:44:02 2014 -0700

----------------------------------------------------------------------
 .../main/scala/org/apache/spark/scheduler/DAGScheduler.scala    | 5 +++++
 1 file changed, 5 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/3bd32f02/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
index d15aaa3..a9fd7e7 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
@@ -26,6 +26,7 @@ import scala.concurrent.Await
 import scala.concurrent.duration._
 import scala.language.postfixOps
 import scala.reflect.ClassTag
+import scala.util.control.NonFatal
 
 import akka.actor._
 import akka.actor.OneForOneStrategy
@@ -771,6 +772,10 @@ class DAGScheduler(
           abortStage(stage, "Task not serializable: " + e.toString)
           runningStages -= stage
           return
+        case NonFatal(e) => // Other exceptions, such as IllegalArgumentException from
Kryo.
+          abortStage(stage, s"Task serialization failed: $e\n${e.getStackTraceString}")
+          runningStages -= stage
+          return
       }
 
       logInfo("Submitting " + tasks.size + " missing tasks from " + stage + " (" + stage.rdd
+ ")")


Mime
View raw message