spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zsxw...@apache.org
Subject spark git commit: [SPARK-25163][SQL] Fix flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuiteCheck
Date Wed, 22 Aug 2018 21:17:09 GMT
Repository: spark
Updated Branches:
  refs/heads/master 310632498 -> 49a1993b1


[SPARK-25163][SQL] Fix flaky test: o.a.s.util.collection.ExternalAppendOnlyMapSuiteCheck

## What changes were proposed in this pull request?

`ExternalAppendOnlyMapSuiteCheck` test is flaky.

We use a `SparkListener` to collect spill metrics of completed stages. `withListener` runs
the code that does spill. Spill status was checked after the code finishes but it was still
in `withListener`. At that time it was possibly not all events to the listener bus are processed.

We should check spill status after all events are processed.

## How was this patch tested?

Locally ran unit tests.

Closes #22181 from viirya/SPARK-25163.

Authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Signed-off-by: Shixiong Zhu <zsxwing@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/49a1993b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/49a1993b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/49a1993b

Branch: refs/heads/master
Commit: 49a1993b168accb6f188c682546f12ea568173c4
Parents: 3106324
Author: Liang-Chi Hsieh <viirya@gmail.com>
Authored: Wed Aug 22 14:17:05 2018 -0700
Committer: Shixiong Zhu <zsxwing@gmail.com>
Committed: Wed Aug 22 14:17:05 2018 -0700

----------------------------------------------------------------------
 core/src/main/scala/org/apache/spark/TestUtils.scala | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/49a1993b/core/src/main/scala/org/apache/spark/TestUtils.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/TestUtils.scala b/core/src/main/scala/org/apache/spark/TestUtils.scala
index 6cc8fe1..c2ebd38 100644
--- a/core/src/main/scala/org/apache/spark/TestUtils.scala
+++ b/core/src/main/scala/org/apache/spark/TestUtils.scala
@@ -173,10 +173,11 @@ private[spark] object TestUtils {
    * Run some code involving jobs submitted to the given context and assert that the jobs
spilled.
    */
   def assertSpilled(sc: SparkContext, identifier: String)(body: => Unit): Unit = {
-    withListener(sc, new SpillListener) { listener =>
+    val listener = new SpillListener
+    withListener(sc, listener) { _ =>
       body
-      assert(listener.numSpilledStages > 0, s"expected $identifier to spill, but did not")
     }
+    assert(listener.numSpilledStages > 0, s"expected $identifier to spill, but did not")
   }
 
   /**
@@ -184,10 +185,11 @@ private[spark] object TestUtils {
    * did not spill.
    */
   def assertNotSpilled(sc: SparkContext, identifier: String)(body: => Unit): Unit = {
-    withListener(sc, new SpillListener) { listener =>
+    val listener = new SpillListener
+    withListener(sc, listener) { _ =>
       body
-      assert(listener.numSpilledStages == 0, s"expected $identifier to not spill, but did")
     }
+    assert(listener.numSpilledStages == 0, s"expected $identifier to not spill, but did")
   }
 
   /**


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message