From issues-return-292611-archive-asf-public=cust-asf.ponee.io@spark.apache.org Wed Jul 21 10:00:24 2021 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mxout1-he-de.apache.org (mxout1-he-de.apache.org [95.216.194.37]) by mx-eu-01.ponee.io (Postfix) with ESMTPS id F129A18064E for ; Wed, 21 Jul 2021 12:00:23 +0200 (CEST) Received: from mail.apache.org (mailroute1-lw-us.apache.org [207.244.88.153]) by mxout1-he-de.apache.org (ASF Mail Server at mxout1-he-de.apache.org) with SMTP id BB9B46238E for ; Wed, 21 Jul 2021 10:00:22 +0000 (UTC) Received: (qmail 85024 invoked by uid 500); 21 Jul 2021 10:00:21 -0000 Mailing-List: contact issues-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@spark.apache.org Received: (qmail 84997 invoked by uid 99); 21 Jul 2021 10:00:21 -0000 Received: from mailrelay1-he-de.apache.org (HELO mailrelay1-he-de.apache.org) (116.203.21.61) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Jul 2021 10:00:21 +0000 Received: from jira2-he-de.apache.org (jira2-he-de.apache.org [168.119.33.54]) by mailrelay1-he-de.apache.org (ASF Mail Server at mailrelay1-he-de.apache.org) with ESMTPS id EF9CD3E8CA for ; Wed, 21 Jul 2021 10:00:20 +0000 (UTC) Received: from jira2-he-de.apache.org (localhost.localdomain [127.0.0.1]) by jira2-he-de.apache.org (ASF Mail Server at jira2-he-de.apache.org) with ESMTP id D5CE6C80370 for ; Wed, 21 Jul 2021 10:00:20 +0000 (UTC) Date: Wed, 21 Jul 2021 10:00:20 +0000 (UTC) From: "Apache Spark (Jira)" To: issues@spark.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (SPARK-36242) Ensure spill file closed before set success to true in ExternalSorter.spillMemoryIteratorToDisk method MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/SPARK-36242?page=3Dcom.atlassia= n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D173= 84803#comment-17384803 ]=20 Apache Spark commented on SPARK-36242: -------------------------------------- User 'LuciferYang' has created a pull request for this issue: https://github.com/apache/spark/pull/33460 > Ensure spill file closed before set success to true in ExternalSorter.spi= llMemoryIteratorToDisk method > -------------------------------------------------------------------------= ----------------------------- > > Key: SPARK-36242 > URL: https://issues.apache.org/jira/browse/SPARK-36242 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 3.3.0 > Reporter: Yang Jie > Priority: Minor > > The processes of=C2=A0ExternalSorter.spillMemoryIteratorToDisk and=C2=A0E= xternalAppendOnlyMap.spillMemoryIteratorToDisk are similar,=C2=A0but there = are some differences in setting `success =3D true` > =C2=A0 > Code of ExternalSorter.spillMemoryIteratorToDisk as follows: > =C2=A0 > {code:java} > if (objectsWritten > 0) { > flush() > } else { > writer.revertPartialWritesAndClose() > } > success =3D true > } finally { > if (success) { > writer.close() > } else { > ... > } > }{code} > Code of ExternalSorter.spillMemoryIteratorToDisk as follows: > {code:java} > if (objectsWritten > 0) { > flush() > writer.close() > } else { > writer.revertPartialWritesAndClose() > } > success =3D true > } finally { > if (!success) { > ... > } > }{code} > It seems that the processing of `ExternalSorter.spillMemoryIteratorToDisk= ` mehod is more reasonable, We should make sure setting `success =3D true` = after the spill file is closed > =C2=A0 -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org For additional commands, e-mail: issues-help@spark.apache.org