spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andrewo...@apache.org
Subject spark git commit: [SPARK-11649] Properly set Akka frame size in SparkListenerSuite test
Date Wed, 18 Nov 2015 23:55:47 GMT
Repository: spark
Updated Branches:
  refs/heads/master 7e987de17 -> 3a9851936


[SPARK-11649] Properly set Akka frame size in SparkListenerSuite test

SparkListenerSuite's _"onTaskGettingResult() called when result fetched remotely"_ test was
extremely slow (1 to 4 minutes to run) and recently became extremely flaky, frequently failing
with OutOfMemoryError.

The root cause was the fact that this was using `System.setProperty` to set the Akka frame
size, which was not actually modifying the frame size. As a result, this test would allocate
much more data than necessary. The fix here is to simply use SparkConf in order to configure
the frame size.

Author: Josh Rosen <joshrosen@databricks.com>

Closes #9822 from JoshRosen/SPARK-11649.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3a985193
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3a985193
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3a985193

Branch: refs/heads/master
Commit: 3a9851936ddfe5bcb6a7f364d535fac977551f5d
Parents: 7e987de
Author: Josh Rosen <joshrosen@databricks.com>
Authored: Wed Nov 18 15:55:41 2015 -0800
Committer: Andrew Or <andrew@databricks.com>
Committed: Wed Nov 18 15:55:41 2015 -0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/scheduler/SparkListenerSuite.scala   | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/3a985193/core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala
----------------------------------------------------------------------
diff --git a/core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala b/core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala
index 53102b9..84e5458 100644
--- a/core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala
+++ b/core/src/test/scala/org/apache/spark/scheduler/SparkListenerSuite.scala
@@ -269,14 +269,15 @@ class SparkListenerSuite extends SparkFunSuite with LocalSparkContext
with Match
   }
 
   test("onTaskGettingResult() called when result fetched remotely") {
-    sc = new SparkContext("local", "SparkListenerSuite")
+    val conf = new SparkConf().set("spark.akka.frameSize", "1")
+    sc = new SparkContext("local", "SparkListenerSuite", conf)
     val listener = new SaveTaskEvents
     sc.addSparkListener(listener)
 
     // Make a task whose result is larger than the akka frame size
-    System.setProperty("spark.akka.frameSize", "1")
     val akkaFrameSize =
       sc.env.actorSystem.settings.config.getBytes("akka.remote.netty.tcp.maximum-frame-size").toInt
+    assert(akkaFrameSize === 1024 * 1024)
     val result = sc.parallelize(Seq(1), 1)
       .map { x => 1.to(akkaFrameSize).toArray }
       .reduce { case (x, y) => x }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message