spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: [SPARK-5759][Yarn]ExecutorRunnable should catch YarnException while NMClient start contain...
Date Thu, 12 Feb 2015 22:51:16 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 11d108030 -> 02d5b32bb


[SPARK-5759][Yarn]ExecutorRunnable should catch YarnException while NMClient start contain...

some time since some reasons, it lead to some exception while NMClient start some containers.example:we
do not config spark_shuffle on some machines, so it will throw a exception:
java.lang.Error: org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:spark_shuffle
does not exist.
because YarnAllocator use ThreadPoolExecutor to start Container, so we can not find which
container or hostname throw exception. I think we should catch YarnException in ExecutorRunnable
when start container. if there are some exceptions, we can know the container id or hostname
of failed container.

Author: lianhuiwang <lianhuiwang09@gmail.com>

Closes #4554 from lianhuiwang/SPARK-5759 and squashes the following commits:

caf5a99 [lianhuiwang] use SparkException to warp exception
c02140f [lianhuiwang] ExecutorRunnable should catch YarnException while NMClient start container

(cherry picked from commit 947b8bd82ec0f4c45910e6d781df4661f56e4587)
Signed-off-by: Andrew Or <andrew@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/02d5b32b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/02d5b32b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/02d5b32b

Branch: refs/heads/branch-1.3
Commit: 02d5b32bbebc055c1b4cde4f08a8194397921aa9
Parents: 11d1080
Author: lianhuiwang <lianhuiwang09@gmail.com>
Authored: Thu Feb 12 14:50:16 2015 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Thu Feb 12 14:51:15 2015 -0800

----------------------------------------------------------------------
 .../org/apache/spark/deploy/yarn/ExecutorRunnable.scala   | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/02d5b32b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
----------------------------------------------------------------------
diff --git a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
index 6d5b8fd..c1d3f73 100644
--- a/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
+++ b/yarn/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala
@@ -38,7 +38,7 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration
 import org.apache.hadoop.yarn.ipc.YarnRPC
 import org.apache.hadoop.yarn.util.{ConverterUtils, Records}
 
-import org.apache.spark.{SecurityManager, SparkConf, Logging}
+import org.apache.spark.{Logging, SecurityManager, SparkConf, SparkException}
 import org.apache.spark.network.util.JavaUtils
 
 class ExecutorRunnable(
@@ -109,7 +109,13 @@ class ExecutorRunnable(
     }
 
     // Send the start request to the ContainerManager
-    nmClient.startContainer(container, ctx)
+    try {
+      nmClient.startContainer(container, ctx)
+    } catch {
+      case ex: Exception =>
+        throw new SparkException(s"Exception while starting container ${container.getId}"
+
+          s" on host $hostname", ex)
+    }
   }
 
   private def prepareCommand(


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message