spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hvanhov...@apache.org
Subject spark git commit: [SPARK-22721] BytesToBytesMap peak memory not updated.
Date Fri, 08 Dec 2017 11:19:49 GMT
Repository: spark
Updated Branches:
  refs/heads/master f88a67bf0 -> f28b1a4c4


[SPARK-22721] BytesToBytesMap peak memory not updated.

## What changes were proposed in this pull request?

Follow-up to earlier commit.
The peak memory of BytesToBytesMap is not updated in more places - spill() and destructiveIterator().

## How was this patch tested?

Manually.

Author: Juliusz Sompolski <julek@databricks.com>

Closes #19923 from juliuszsompolski/SPARK-22721cd.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f28b1a4c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f28b1a4c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f28b1a4c

Branch: refs/heads/master
Commit: f28b1a4c41d41e069b882bac06a68b2c45eb156c
Parents: f88a67b
Author: Juliusz Sompolski <julek@databricks.com>
Authored: Fri Dec 8 12:19:45 2017 +0100
Committer: Herman van Hovell <hvanhovell@databricks.com>
Committed: Fri Dec 8 12:19:45 2017 +0100

----------------------------------------------------------------------
 .../main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java    | 3 +++
 1 file changed, 3 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/spark/blob/f28b1a4c/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
----------------------------------------------------------------------
diff --git a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
index 7fdcf22..5f00455 100644
--- a/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
+++ b/core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
@@ -347,6 +347,8 @@ public final class BytesToBytesMap extends MemoryConsumer {
           return 0L;
         }
 
+        updatePeakMemoryUsed();
+
         // TODO: use existing ShuffleWriteMetrics
         ShuffleWriteMetrics writeMetrics = new ShuffleWriteMetrics();
 
@@ -424,6 +426,7 @@ public final class BytesToBytesMap extends MemoryConsumer {
    * `lookup()`, the behavior of the returned iterator is undefined.
    */
   public MapIterator destructiveIterator() {
+    updatePeakMemoryUsed();
     return new MapIterator(numValues, loc, true);
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org


Mime
View raw message