spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-6133] Make sc.stop() idempotent
Date Fri, 13 Mar 2015 18:33:42 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 338bea7b3 -> a08588c7e


[SPARK-6133] Make sc.stop() idempotent

Before we would get the following (benign) error if we called `sc.stop()` twice. This is because
the listener bus would try to post the end event again even after it has already stopped.
This happens occasionally when flaky tests fail, usually as a result of other sources of error.
Either way we shouldn't be logging this error when it is not the cause of the failure.
```
ERROR LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerApplicationEnd(1425348445682)
```

Author: Andrew Or <andrew@databricks.com>

Closes #4871 from andrewor14/sc-stop and squashes the following commits:

a14afc5 [Andrew Or] Move code after code
915db16 [Andrew Or] Move code into code


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a08588c7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a08588c7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a08588c7

Branch: refs/heads/branch-1.3
Commit: a08588c7eeaecf7003073c092320b37abd166191
Parents: 338bea7
Author: Andrew Or <andrew@databricks.com>
Authored: Tue Mar 3 15:09:57 2015 -0800
Committer: Sean Owen <sowen@cloudera.com>
Committed: Fri Mar 13 18:33:27 2015 +0000

----------------------------------------------------------------------
 core/src/main/scala/org/apache/spark/SparkContext.scala | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/a08588c7/core/src/main/scala/org/apache/spark/SparkContext.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala
index f80338e..023d54e 100644
--- a/core/src/main/scala/org/apache/spark/SparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/SparkContext.scala
@@ -1369,10 +1369,10 @@ class SparkContext(config: SparkConf) extends Logging with ExecutorAllocationCli
   /** Shut down the SparkContext. */
   def stop() {
     SparkContext.SPARK_CONTEXT_CONSTRUCTOR_LOCK.synchronized {
-      postApplicationEnd()
-      ui.foreach(_.stop())
       if (!stopped) {
         stopped = true
+        postApplicationEnd()
+        ui.foreach(_.stop())
         env.metricsSystem.report()
         metadataCleaner.cancel()
         cleaner.foreach(_.stop())


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message