From commits-return-30266-archive-asf-public=cust-asf.ponee.io@spark.apache.org Fri Feb 9 15:45:19 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id CAD44180654 for ; Fri, 9 Feb 2018 15:45:19 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id BA8BF160C4C; Fri, 9 Feb 2018 14:45:19 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0CE67160C2E for ; Fri, 9 Feb 2018 15:45:18 +0100 (CET) Received: (qmail 23446 invoked by uid 500); 9 Feb 2018 14:45:18 -0000 Mailing-List: contact commits-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list commits@spark.apache.org Received: (qmail 23437 invoked by uid 99); 9 Feb 2018 14:45:18 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 09 Feb 2018 14:45:18 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 244C7DFAF3; Fri, 9 Feb 2018 14:45:18 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: srowen@apache.org To: commits@spark.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: spark git commit: [SPARK-23358][CORE] When the number of partitions is greater than 2^28, it will result in an error result Date: Fri, 9 Feb 2018 14:45:18 +0000 (UTC) Repository: spark Updated Branches: refs/heads/branch-2.3 196304a3a -> 08eb95f60 [SPARK-23358][CORE] When the number of partitions is greater than 2^28, it will result in an error result ## What changes were proposed in this pull request? In the `checkIndexAndDataFile`,the `blocks` is the ` Int` type, when it is greater than 2^28, `blocks*8` will overflow, and this will result in an error result. In fact, `blocks` is actually the number of partitions. ## How was this patch tested? Manual test Author: liuxian Closes #20544 from 10110346/overflow. (cherry picked from commit f77270b8811bbd8956d0c08fa556265d2c5ee20e) Signed-off-by: Sean Owen Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/08eb95f6 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/08eb95f6 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/08eb95f6 Branch: refs/heads/branch-2.3 Commit: 08eb95f609f5d356c89dedcefa768b12a7a8b96c Parents: 196304a Author: liuxian Authored: Fri Feb 9 08:45:06 2018 -0600 Committer: Sean Owen Committed: Fri Feb 9 08:45:15 2018 -0600 ---------------------------------------------------------------------- .../scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/08eb95f6/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala ---------------------------------------------------------------------- diff --git a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala index 266ee42..2414b94 100644 --- a/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala +++ b/core/src/main/scala/org/apache/spark/shuffle/IndexShuffleBlockResolver.scala @@ -84,7 +84,7 @@ private[spark] class IndexShuffleBlockResolver( */ private def checkIndexAndDataFile(index: File, data: File, blocks: Int): Array[Long] = { // the index file should have `block + 1` longs as offset. - if (index.length() != (blocks + 1) * 8) { + if (index.length() != (blocks + 1) * 8L) { return null } val lengths = new Array[Long](blocks) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org For additional commands, e-mail: commits-help@spark.apache.org