spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jianfei Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-17562) I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk
Date Sun, 18 Sep 2016 02:11:20 GMT

    [ https://issues.apache.org/jira/browse/SPARK-17562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15500059#comment-15500059
] 

Jianfei Wang commented on SPARK-17562:
--------------------------------------

[~joshrosen] please check this. thank you very much!

> I think a little code is unnecessary to exist in ExternalSorter.spillMemoryIteratorToDisk
> -----------------------------------------------------------------------------------------
>
>                 Key: SPARK-17562
>                 URL: https://issues.apache.org/jira/browse/SPARK-17562
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.0.0
>            Reporter: Jianfei Wang
>            Priority: Trivial
>              Labels: easyfix, performance
>
> In ExternalSorter.spillMemoryIteratorToDisk, I think the code below will never be executed,
so we can remove them
> {code}
> else  {
> writer.revertPartialWritersAndClose()
> }
> {code}
> the source code is as below:
> {code}
> try {
>       while (inMemoryIterator.hasNext) {
>         val partitionId = inMemoryIterator.nextPartition()
>         require(partitionId >= 0 && partitionId < numPartitions,
>           s"partition Id: ${partitionId} should be in the range [0, ${numPartitions})")
>         inMemoryIterator.writeNext(writer)
>         elementsPerPartition(partitionId) += 1
>         objectsWritten += 1
>         if (objectsWritten == serializerBatchSize) {
>           flush()
>         }
>       }
>       if (objectsWritten > 0) {
>         flush()
>       } else {
>         writer.revertPartialWritesAndClose()
>    }
>       success = true
>     } finally {
>       if (success) {
>         writer.close()
>       } else {
>         // This code path only happens if an exception was thrown above before we set
success;
>         // close our stuff and let the exception be thrown further
>         writer.revertPartialWritesAndClose()
>         if (file.exists()) {
>           if (!file.delete()) {
>             logWarning(s"Error deleting ${file}")
>           }
>         }
>       }
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message