spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sital Kedia (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-17839) UnsafeSorterSpillReader should use Nio's directbuffer to read the spill files in order to avoid additional copy
Date Sun, 09 Oct 2016 15:36:20 GMT
Sital Kedia created SPARK-17839:
-----------------------------------

             Summary: UnsafeSorterSpillReader should use Nio's directbuffer to read the spill
files in order to avoid additional copy
                 Key: SPARK-17839
                 URL: https://issues.apache.org/jira/browse/SPARK-17839
             Project: Spark
          Issue Type: Improvement
          Components: Shuffle
    Affects Versions: 2.0.1
            Reporter: Sital Kedia
            Priority: Minor


Currently we use BufferedInputStream to read the shuffle file which copies the file content
from os buffer cache to the user buffer. This adds additional latency in reading the spill
files. We made a change to use java nio's direct buffer to read the spill files and for certain
jobs spilling significant amount of data, we see between 5 - 7% speedup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message