spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting killed on yarn restart.
Date Fri, 06 Feb 2015 21:47:44 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.2 36f70de83 -> d89964f86


SPARK-5613: Catch the ApplicationNotFoundException exception to avoid thread from getting
killed on yarn restart.

[SPARK-5613] Added a  catch block to catch the ApplicationNotFoundException. Without this
catch block the thread gets killed on occurrence of this exception. This Exception occurs
when yarn restarts and tries to find an application id for a spark job which got interrupted
due to yarn getting stopped.
See the stacktrace in the bug for more details.

Author: Kashish Jain <kashish.jain@guavus.com>

Closes #4392 from kasjain/branch-1.2 and squashes the following commits:

4831000 [Kashish Jain] SPARK-5613: Catch the ApplicationNotFoundException exception to avoid
thread from getting killed on yarn restart.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d89964f8
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d89964f8
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d89964f8

Branch: refs/heads/branch-1.2
Commit: d89964f86e288dbdd67e08f4eb97f374c4d437f3
Parents: 36f70de
Author: Kashish Jain <kashish.jain@guavus.com>
Authored: Fri Feb 6 13:47:23 2015 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Fri Feb 6 13:47:23 2015 -0800

----------------------------------------------------------------------
 .../scheduler/cluster/YarnClientSchedulerBackend.scala   | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/d89964f8/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
----------------------------------------------------------------------
diff --git a/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
b/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
index 09597bd..15f8049 100644
--- a/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
+++ b/yarn/common/src/main/scala/org/apache/spark/scheduler/cluster/YarnClientSchedulerBackend.scala
@@ -20,6 +20,7 @@ package org.apache.spark.scheduler.cluster
 import scala.collection.mutable.ArrayBuffer
 
 import org.apache.hadoop.yarn.api.records.{ApplicationId, YarnApplicationState}
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException
 
 import org.apache.spark.{SparkException, Logging, SparkContext}
 import org.apache.spark.deploy.yarn.{Client, ClientArguments}
@@ -132,8 +133,14 @@ private[spark] class YarnClientSchedulerBackend(
     val t = new Thread {
       override def run() {
         while (!stopping) {
-          val report = client.getApplicationReport(appId)
-          val state = report.getYarnApplicationState()
+          var state : YarnApplicationState = null
+          try {
+            val report = client.getApplicationReport(appId)
+            state = report.getYarnApplicationState()
+          } catch {
+            case e : ApplicationNotFoundException =>
+              state = YarnApplicationState.KILLED
+          }
           if (state == YarnApplicationState.FINISHED ||
             state == YarnApplicationState.KILLED ||
             state == YarnApplicationState.FAILED) {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message