spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-10546] Check partitionId's range in ExternalSorter#spill()
Date Fri, 11 Sep 2015 20:45:50 GMT
Repository: spark
Updated Branches:
  refs/heads/master 5f4644476 -> b231ab893


[SPARK-10546] Check partitionId's range in ExternalSorter#spill()

See this thread for background:
http://search-hadoop.com/m/q3RTt0rWvIkHAE81

We should check the range of partition Id and provide meaningful message through exception.

Alternatively, we can use abs() and modulo to force the partition Id into legitimate range.
However, expectation is that user should correct the logic error in his / her code.

Author: tedyu <yuzhihong@gmail.com>

Closes #8703 from tedyu/master.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b231ab89
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b231ab89
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b231ab89

Branch: refs/heads/master
Commit: b231ab8938ae3c4fc2089cfc69c0d8164807d533
Parents: 5f46444
Author: tedyu <yuzhihong@gmail.com>
Authored: Fri Sep 11 21:45:45 2015 +0100
Committer: Sean Owen <sowen@cloudera.com>
Committed: Fri Sep 11 21:45:45 2015 +0100

----------------------------------------------------------------------
 .../scala/org/apache/spark/util/collection/ExternalSorter.scala    | 2 ++
 1 file changed, 2 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/b231ab89/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
index 138c05d..31230d5 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/ExternalSorter.scala
@@ -297,6 +297,8 @@ private[spark] class ExternalSorter[K, V, C](
       val it = collection.destructiveSortedWritablePartitionedIterator(comparator)
       while (it.hasNext) {
         val partitionId = it.nextPartition()
+        require(partitionId >= 0 && partitionId < numPartitions,
+          s"partition Id: ${partitionId} should be in the range [0, ${numPartitions})")
         it.writeNext(writer)
         elementsPerPartition(partitionId) += 1
         objectsWritten += 1


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message