spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wenc...@apache.org
Subject spark git commit: [SPARK-23962][SQL][TEST] Fix race in currentExecutionIds().
Date Thu, 12 Apr 2018 07:58:42 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.3 03a4dfd69 -> 571269519


[SPARK-23962][SQL][TEST] Fix race in currentExecutionIds().

SQLMetricsTestUtils.currentExecutionIds() was racing with the listener
bus, which lead to some flaky tests.  We should wait till the listener bus is
empty.

I tested by adding some Thread.sleep()s in SQLAppStatusListener, which
reproduced the exceptions I saw on Jenkins.  With this change, they went
away.

Author: Imran Rashid <irashid@cloudera.com>

Closes #21041 from squito/SPARK-23962.

(cherry picked from commit 6a2289ecf020a99cd9b3bcea7da5e78fb4e0303a)
Signed-off-by: Wenchen Fan <wenchen@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/57126951
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/57126951
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/57126951

Branch: refs/heads/branch-2.3
Commit: 571269519a49e0651575de18de81b788b6548afd
Parents: 03a4dfd
Author: Imran Rashid <irashid@cloudera.com>
Authored: Thu Apr 12 15:58:04 2018 +0800
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Thu Apr 12 15:58:36 2018 +0800

----------------------------------------------------------------------
 .../org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala | 1 +
 1 file changed, 1 insertion(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/57126951/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
index 122d287..e22bbb6 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/SQLMetricsTestUtils.scala
@@ -34,6 +34,7 @@ trait SQLMetricsTestUtils extends SQLTestUtils {
   import testImplicits._
 
   protected def currentExecutionIds(): Set[Long] = {
+    spark.sparkContext.listenerBus.waitUntilEmpty(10000)
     statusStore.executionsList.map(_.executionId).toSet
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message