spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zsxw...@apache.org
Subject spark git commit: [SPARK-17318][TESTS] Fix ReplSuite replicating blocks of object with class defined in repl again
Date Thu, 01 Sep 2016 06:25:35 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.0 8711b451d -> 6281b74b6


[SPARK-17318][TESTS] Fix ReplSuite replicating blocks of object with class defined in repl
again

## What changes were proposed in this pull request?

After digging into the logs, I noticed the failure is because in this test, it starts a local
cluster with 2 executors. However, when SparkContext is created, executors may be still not
up. When one of the executor is not up during running the job, the blocks won't be replicated.

This PR just adds a wait loop before running the job to fix the flaky test.

## How was this patch tested?

Jenkins

Author: Shixiong Zhu <shixiong@databricks.com>

Closes #14905 from zsxwing/SPARK-17318-2.

(cherry picked from commit 21c0a4fe9d8e21819ba96e7dc2b1f2999d3299ae)
Signed-off-by: Shixiong Zhu <shixiong@databricks.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6281b74b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6281b74b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6281b74b

Branch: refs/heads/branch-2.0
Commit: 6281b74b6965ffcd0600844cea168cbe71ca8248
Parents: 8711b45
Author: Shixiong Zhu <shixiong@databricks.com>
Authored: Wed Aug 31 23:25:20 2016 -0700
Committer: Shixiong Zhu <shixiong@databricks.com>
Committed: Wed Aug 31 23:25:27 2016 -0700

----------------------------------------------------------------------
 .../src/test/scala/org/apache/spark/repl/ReplSuite.scala    | 9 +++++++++
 1 file changed, 9 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/6281b74b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
----------------------------------------------------------------------
diff --git a/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
index f1284b1..f7d7a4f 100644
--- a/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
+++ b/repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala
@@ -399,6 +399,15 @@ class ReplSuite extends SparkFunSuite {
   test("replicating blocks of object with class defined in repl") {
     val output = runInterpreter("local-cluster[2,1,1024]",
       """
+        |val timeout = 60000 // 60 seconds
+        |val start = System.currentTimeMillis
+        |while(sc.getExecutorStorageStatus.size != 3 &&
+        |    (System.currentTimeMillis - start) < timeout) {
+        |  Thread.sleep(10)
+        |}
+        |if (System.currentTimeMillis - start >= timeout) {
+        |  throw new java.util.concurrent.TimeoutException("Executors were not up in 60 seconds")
+        |}
         |import org.apache.spark.storage.StorageLevel._
         |case class Foo(i: Int)
         |val ret = sc.parallelize((1 to 100).map(Foo), 10).persist(MEMORY_AND_DISK_2)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message