From reviews-return-930713-archive-asf-public=cust-asf.ponee.io@spark.apache.org Sat Sep 28 19:40:38 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 4F095180645 for ; Sat, 28 Sep 2019 21:40:38 +0200 (CEST) Received: (qmail 47600 invoked by uid 500); 28 Sep 2019 19:40:37 -0000 Mailing-List: contact reviews-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list reviews@spark.apache.org Received: (qmail 47589 invoked by uid 99); 28 Sep 2019 19:40:37 -0000 Received: from ec2-52-202-80-70.compute-1.amazonaws.com (HELO gitbox.apache.org) (52.202.80.70) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Sep 2019 19:40:37 +0000 From: GitBox To: reviews@spark.apache.org Subject: [GitHub] [spark] JoshRosen commented on a change in pull request #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again Message-ID: <156969963761.24520.14116102993916330151.gitbox@gitbox.apache.org> Date: Sat, 28 Sep 2019 19:40:37 -0000 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit JoshRosen commented on a change in pull request #25953: [SPARK-29244][Core] Prevent freed page in BytesToBytesMap free again URL: https://github.com/apache/spark/pull/25953#discussion_r329324260 ########## File path: core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java ########## @@ -787,8 +783,16 @@ private void allocate(int capacity) { assert (capacity >= 0); capacity = Math.max((int) Math.min(MAX_CAPACITY, ByteArrayMethods.nextPowerOf2(capacity)), 64); assert (capacity <= MAX_CAPACITY); - longArray = allocateArray(capacity * 2L); - longArray.zeroOut(); + try { + longArray = allocateArray(capacity * 2L); + longArray.zeroOut(); + } catch (SparkOutOfMemoryError e) { + // When OOM, allocated page was already freed by `TaskMemoryManager`. + // We should not keep it in `longArray`. Otherwise it might be freed again in task + // complete listeners and cause unnecessary error. + longArray = null; Review comment: In retrospect, with a few years of hindsight, I think there's definitely room to write `BytesToBytesMap` in an easier-to-understand manner: it's quite possible that a different design of helper methods and management of side-effects would prevent this class of bugs or make them easier to fix. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org For additional commands, e-mail: reviews-help@spark.apache.org