spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwendell <...@git.apache.org>
Subject [GitHub] spark pull request: [SPARK-1244] Throw exception if map output sta...
Date Sat, 15 Mar 2014 05:33:47 GMT
Github user pwendell commented on a diff in the pull request:

    https://github.com/apache/spark/pull/147#discussion_r10633600
  
    --- Diff: core/src/test/scala/org/apache/spark/MapOutputTrackerSuite.scala ---
    @@ -136,4 +123,47 @@ class MapOutputTrackerSuite extends FunSuite with LocalSparkContext
{
         // failure should be cached
         intercept[FetchFailedException] { slaveTracker.getServerStatuses(10, 0) }
       }
    +
    +  test("remote fetch exceeding akka frame size") {
    +    val newConf = new SparkConf
    +    newConf.set("spark.akka.frameSize", "1")
    +    newConf.set("spark.akka.askTimeout", "1") // Fail fast
    +    val (masterTracker, slaveTracker) = setUpMasterSlaveSystem(newConf)
    +
    +    // Frame size should be ~123B, and no exception should be thrown
    +    masterTracker.registerShuffle(10, 1)
    +    masterTracker.registerMapOutput(10, 0, new MapStatus(
    +      BlockManagerId("88", "mph", 1000, 0), Array.fill[Byte](10)(0)))
    +    slaveTracker.getServerStatuses(10, 0)
    +
    +    // Frame size should be ~1.1MB, and MapOutputTrackerMasterActor should throw exception
    +    masterTracker.registerShuffle(20, 100)
    +    (0 until 100).foreach { i =>
    +      masterTracker.registerMapOutput(20, i, new MapStatus(
    +        BlockManagerId("999", "mps", 1000, 0), Array.fill[Byte](4000000)(0)))
    +    }
    +    intercept[SparkException] { slaveTracker.getServerStatuses(20, 0) }
    +  }
    +
    +  private def setUpMasterSlaveSystem(conf: SparkConf) = {
    --- End diff --
    
    Setting up an entire actor system in the tests causes this test to take a long time (~4
seconds) and also brings a lot of other code into scope during the tests. What about instead
creating local actor refs for testing? Akka actually has good support for this. You can look
at an example I added here:
    
    https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/deploy/worker/WorkerWatcherSuite.scala#L42
    
    Would that be sufficient to test this case?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message