spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wenc...@apache.org
Subject spark git commit: [SPARK-15354][FLAKY-TEST] TopologyAwareBlockReplicationPolicyBehavior.Peers in 2 racks
Date Thu, 13 Apr 2017 00:38:29 GMT
Repository: spark
Updated Branches:
  refs/heads/master 924c42477 -> a7b430b57


[SPARK-15354][FLAKY-TEST] TopologyAwareBlockReplicationPolicyBehavior.Peers in 2 racks

## What changes were proposed in this pull request?

`TopologyAwareBlockReplicationPolicyBehavior.Peers in 2 racks` is failing occasionally: https://spark-tests.appspot.com/test-details?suite_name=org.apache.spark.storage.TopologyAwareBlockReplicationPolicyBehavior&test_name=Peers+in+2+racks.

This is because, when we generate 10 block manager id to test, they may all belong to the
same rack, as the rack is randomly picked. This PR fixes this problem by forcing each rack
to be picked at least once.

## How was this patch tested?

N/A

Author: Wenchen Fan <wenchen@databricks.com>

Closes #17624 from cloud-fan/test.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a7b430b5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a7b430b5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a7b430b5

Branch: refs/heads/master
Commit: a7b430b5717e263c1fbb55114deca6028ea9c3b3
Parents: 924c424
Author: Wenchen Fan <wenchen@databricks.com>
Authored: Thu Apr 13 08:38:24 2017 +0800
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Thu Apr 13 08:38:24 2017 +0800

----------------------------------------------------------------------
 .../spark/storage/BlockReplicationPolicySuite.scala    | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/a7b430b5/core/src/test/scala/org/apache/spark/storage/BlockReplicationPolicySuite.scala
----------------------------------------------------------------------
diff --git a/core/src/test/scala/org/apache/spark/storage/BlockReplicationPolicySuite.scala
b/core/src/test/scala/org/apache/spark/storage/BlockReplicationPolicySuite.scala
index ecad0f5..dfecd04 100644
--- a/core/src/test/scala/org/apache/spark/storage/BlockReplicationPolicySuite.scala
+++ b/core/src/test/scala/org/apache/spark/storage/BlockReplicationPolicySuite.scala
@@ -70,9 +70,18 @@ class RandomBlockReplicationPolicyBehavior extends SparkFunSuite
     }
   }
 
+  /**
+   * Returns a sequence of [[BlockManagerId]], whose rack is randomly picked from the given
`racks`.
+   * Note that, each rack will be picked at least once from `racks`, if `count` is greater
or equal
+   * to the number of `racks`.
+   */
   protected def generateBlockManagerIds(count: Int, racks: Seq[String]): Seq[BlockManagerId]
= {
-    (1 to count).map{i =>
-      BlockManagerId(s"Exec-$i", s"Host-$i", 10000 + i, Some(racks(Random.nextInt(racks.size))))
+    val randomizedRacks: Seq[String] = Random.shuffle(
+      racks ++ racks.length.until(count).map(_ => racks(Random.nextInt(racks.length)))
+    )
+
+    (0 until count).map { i =>
+      BlockManagerId(s"Exec-$i", s"Host-$i", 10000 + i, Some(randomizedRacks(i)))
     }
   }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message