spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lix...@apache.org
Subject spark git commit: [SPARK-22254][CORE] Fix the arrayMax in BufferHolder
Date Sat, 04 Nov 2017 06:36:02 GMT
Repository: spark
Updated Branches:
  refs/heads/master 891588660 -> bc1e10103


[SPARK-22254][CORE] Fix the arrayMax in BufferHolder

## What changes were proposed in this pull request?

This PR replaces the old the maximum array size (`Int.MaxValue`) with the new one (`ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`).
This PR also refactor the code to calculate the new array size to easily understand why we
have to use `newSize - 2` for allocating a new array.

## How was this patch tested?

Used the existing test

Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com>

Closes #19650 from kiszk/SPARK-22254.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/bc1e1010
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/bc1e1010
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/bc1e1010

Branch: refs/heads/master
Commit: bc1e101039ae3700eab42e633571256440a42b9d
Parents: 8915886
Author: Kazuaki Ishizaki <ishizaki@jp.ibm.com>
Authored: Fri Nov 3 23:35:57 2017 -0700
Committer: gatorsmile <gatorsmile@gmail.com>
Committed: Fri Nov 3 23:35:57 2017 -0700

----------------------------------------------------------------------
 .../spark/util/collection/CompactBuffer.scala       | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/bc1e1010/core/src/main/scala/org/apache/spark/util/collection/CompactBuffer.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/collection/CompactBuffer.scala b/core/src/main/scala/org/apache/spark/util/collection/CompactBuffer.scala
index f5d2fa1..5d36931 100644
--- a/core/src/main/scala/org/apache/spark/util/collection/CompactBuffer.scala
+++ b/core/src/main/scala/org/apache/spark/util/collection/CompactBuffer.scala
@@ -19,6 +19,8 @@ package org.apache.spark.util.collection
 
 import scala.reflect.ClassTag
 
+import org.apache.spark.unsafe.array.ByteArrayMethods
+
 /**
  * An append-only buffer similar to ArrayBuffer, but more memory-efficient for small buffers.
  * ArrayBuffer always allocates an Object array to store the data, with 16 entries by default,
@@ -126,16 +128,16 @@ private[spark] class CompactBuffer[T: ClassTag] extends Seq[T] with
Serializable
 
   /** Increase our size to newSize and grow the backing array if needed. */
   private def growToSize(newSize: Int): Unit = {
-    // Some JVMs can't allocate arrays of length Integer.MAX_VALUE; actual max is somewhat
-    // smaller. Be conservative and lower the cap a little.
-    val arrayMax = Int.MaxValue - 8
-    if (newSize < 0 || newSize - 2 > arrayMax) {
+    // since two fields are hold in element0 and element1, an array holds newSize - 2 elements
+    val newArraySize = newSize - 2
+    val arrayMax = ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH
+    if (newSize < 0 || newArraySize > arrayMax) {
       throw new UnsupportedOperationException(s"Can't grow buffer past $arrayMax elements")
     }
-    val capacity = if (otherElements != null) otherElements.length + 2 else 2
-    if (newSize > capacity) {
+    val capacity = if (otherElements != null) otherElements.length else 0
+    if (newArraySize > capacity) {
       var newArrayLen = 8L
-      while (newSize - 2 > newArrayLen) {
+      while (newArraySize > newArrayLen) {
         newArrayLen *= 2
       }
       if (newArrayLen > arrayMax) {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message