spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sro...@apache.org
Subject spark git commit: [SPARK-23358][CORE] When the number of partitions is greater than 2^28, it will result in an error result
Date Fri, 09 Feb 2018 14:45:18 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-2.3 196304a3a -> 08eb95f60


[SPARK-23358][CORE] When the number of partitions is greater than 2^28, it will result in
an error result

## What changes were proposed in this pull request?
In the `checkIndexAndDataFile`,the `blocks` is the ` Int` type,  when it is greater than 2^28,
`blocks*8` will overflow, and this will result in an error result.
In fact, `blocks` is actually the number of partitions.

## How was this patch tested?
Manual test

Author: liuxian <liu.xian3@zte.com.cn>

Closes #20544 from 10110346/overflow.

(cherry picked from commit f77270b8811bbd8956d0c08fa556265d2c5ee20e)
Signed-off-by: Sean Owen <sowen@cloudera.com>


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/08eb95f6
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/08eb95f6
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/08eb95f6

Branch: refs/heads/branch-2.3
Commit: 08eb95f609f5d356c89dedcefa768b12a7a8b96c
Parents: 196304a
Author: liuxian <liu.xian3@zte.com.cn>
Authored: Fri Feb 9 08:45:06 2018 -0600
Committer: Sean Owen <sowen@cloudera.com>
Committed: Fri Feb 9 08:45:15 2018 -0600

----------------------------------------------------------------------
 .../scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/08eb95f6/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
index 266ee42..2414b94 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala
@@ -84,7 +84,7 @@ private[spark] class IndexShuffleBlockResolver(
    */
   private def checkIndexAndDataFile(index: File, data: File, blocks: Int): Array[Long] =
{
     // the index file should have `block + 1` longs as offset.
-    if (index.length() != (blocks + 1) * 8) {
+    if (index.length() != (blocks + 1) * 8L) {
       return null
     }
     val lengths = new Array[Long](blocks)


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message