spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wenc...@apache.org
Subject spark git commit: [SPARK-22097][CORE] Request an accurate memory after we unrolled the block
Date Thu, 12 Oct 2017 12:27:01 GMT
Repository: spark
Updated Branches:
  refs/heads/master 274f0efef -> b5c1ef7a8


[SPARK-22097][CORE] Request an accurate memory after we unrolled the block

## What changes were proposed in this pull request?

We only need request `bbos.size - unrollMemoryUsedByThisBlock` after unrolled the block.

## How was this patch tested?

Existing UT.

Author: Xianyang Liu <xianyang.liu@intel.com>

Closes #19316 from ConeyLiu/putIteratorAsBytes.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b5c1ef7a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b5c1ef7a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b5c1ef7a

Branch: refs/heads/master
Commit: b5c1ef7a8e4db4067bc361d10d554ee9a538423f
Parents: 274f0ef
Author: Xianyang Liu <xianyang.liu@intel.com>
Authored: Thu Oct 12 20:26:51 2017 +0800
Committer: Wenchen Fan <wenchen@databricks.com>
Committed: Thu Oct 12 20:26:51 2017 +0800

----------------------------------------------------------------------
 .../scala/org/apache/spark/storage/memory/MemoryStore.scala  | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/b5c1ef7a/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala b/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala
index 651e9c7..17f7a69 100644
--- a/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala
+++ b/core/src/main/scala/org/apache/spark/storage/memory/MemoryStore.scala
@@ -388,7 +388,13 @@ private[spark] class MemoryStore(
     // perform one final call to attempt to allocate additional memory if necessary.
     if (keepUnrolling) {
       serializationStream.close()
-      reserveAdditionalMemoryIfNecessary()
+      if (bbos.size > unrollMemoryUsedByThisBlock) {
+        val amountToRequest = bbos.size - unrollMemoryUsedByThisBlock
+        keepUnrolling = reserveUnrollMemoryForThisTask(blockId, amountToRequest, memoryMode)
+        if (keepUnrolling) {
+          unrollMemoryUsedByThisBlock += amountToRequest
+        }
+      }
     }
 
     if (keepUnrolling) {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message