spark-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [spark] JoshRosen commented on a change in pull request #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again
Date Sat, 28 Sep 2019 19:40:37 GMT
JoshRosen commented on a change in pull request #25953: [SPARK-29244][Core] Prevent freed page
in BytesToBytesMap free again
URL: https://github.com/apache/spark/pull/25953#discussion_r329324260
 
 

 ##########
 File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java
 ##########
 @@ -787,8 +783,16 @@ private void allocate(int capacity) {
     assert (capacity >= 0);
     capacity = Math.max((int) Math.min(MAX_CAPACITY, ByteArrayMethods.nextPowerOf2(capacity)),
64);
     assert (capacity <= MAX_CAPACITY);
-    longArray = allocateArray(capacity * 2L);
-    longArray.zeroOut();
+    try {
+      longArray = allocateArray(capacity * 2L);
+      longArray.zeroOut();
+    } catch (SparkOutOfMemoryError e) {
+      // When OOM, allocated page was already freed by `TaskMemoryManager`.
+      // We should not keep it in `longArray`. Otherwise it might be freed again in task
+      // complete listeners and cause unnecessary error.
+      longArray = null;
 
 Review comment:
   In retrospect, with a few years of hindsight, I think there's definitely room to write
`BytesToBytesMap` in an easier-to-understand manner: it's quite possible that a different
design of helper methods and management of side-effects would prevent this class of bugs or
make them easier to fix.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Mime
View raw message