drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-6002) Avoid memory copy from direct buffer to heap while spilling to local disk
Date Sat, 02 Dec 2017 00:27:11 GMT

    [ https://issues.apache.org/jira/browse/DRILL-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275244#comment-16275244
] 

ASF GitHub Bot commented on DRILL-6002:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1058#discussion_r154477971
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/BatchGroup.java
---
    @@ -280,17 +278,14 @@ private void closeInputStream() throws IOException {
           logger.trace("Summary: Read {} bytes from {}", readLength, path);
         }
     
    -    public long closeOutputStream() throws IOException {
    -      if (outputStream == null) {
    -        return 0;
    +    public void closeWriter() throws IOException {
    +      if (writer == null) {
    +        return;
           }
    -      long writeSize = spillSet.getPosition(outputStream);
    -      spillSet.tallyWriteBytes(writeSize);
    -      outputStream.close();
    -      outputStream = null;
    +      long writeSize = writer.close();
    +      long timeNs = writer.timeNs();
    --- End diff --
    
    See comment above. Updating spill set here decouples the writer from the spill set. The
`BatchGroup` already knows about the spill set.


> Avoid memory copy from direct buffer to heap while spilling to local disk
> -------------------------------------------------------------------------
>
>                 Key: DRILL-6002
>                 URL: https://issues.apache.org/jira/browse/DRILL-6002
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Vlad Rozov
>            Assignee: Vlad Rozov
>
> When spilling to a local disk or to any file system that supports WritableByteChannel
it is preferable to avoid copy from off-heap to java heap as WritableByteChannel can work
directly with the off-heap memory.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message