spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wenc...@apache.org
Subject spark git commit: [SPARK-21159][CORE] Don't try to connect to launcher in standalone cluster mode.
Date Sat, 24 Jun 2017 05:36:14 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 a3088d23a -> 96c04f1ed


[SPARK-21159][CORE] Don't try to connect to launcher in standalone cluster mode.

Monitoring for standalone cluster mode is not implemented (see SPARK-11033), but
the same scheduler implementation is used, and if it tries to connect to the
launcher it will fail. So fix the scheduler so it only tries that in client mode;
cluster mode applications will be correctly launched and will work, but monitoring
through the launcher handle will not be available.

Tested by running a cluster mode app with "SparkLauncher.startApplication".

Author: Marcelo Vanzin <vanzin@cloudera.com>

Closes #18397 from vanzin/SPARK-21159.

(cherry picked from commit bfd73a7c48b87456d1b84d826e04eca938a1be64)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/96c04f1e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/96c04f1e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/96c04f1e

Branch: refs/heads/branch-2.2
Commit: 96c04f1edcd53798d9db5a356482248868a0a905
Parents: a3088d2
Author: Marcelo Vanzin <vanzin@cloudera.com>
Authored: Sat Jun 24 13:23:43 2017 +0800
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Sat Jun 24 13:36:08 2017 +0800

----------------------------------------------------------------------
 .../spark/scheduler/cluster/StandaloneSchedulerBackend.scala | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/96c04f1e/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
b/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
index 0529fe9..22ca14f 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/cluster/StandaloneSchedulerBackend.scala
@@ -58,7 +58,13 @@ private[spark] class StandaloneSchedulerBackend(
 
   override def start() {
     super.start()
-    launcherBackend.connect()
+
+    // SPARK-21159. The scheduler backend should only try to connect to the launcher when
in client
+    // mode. In cluster mode, the code that submits the application to the Master needs to
connect
+    // to the launcher instead.
+    if (sc.deployMode == "client") {
+      launcherBackend.connect()
+    }
 
     // The endpoint for executors to talk to us
     val driverUrl = RpcEndpointAddress(


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message