drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-6002) Avoid memory copy from direct buffer to heap while spilling to local disk
Date Sat, 02 Dec 2017 00:27:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-6002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275240#comment-16275240
] 

ASF GitHub Bot commented on DRILL-6002:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1058#discussion_r154477573
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/spill/SpillSet.java
---
    @@ -331,6 +337,43 @@ public long getReadBytes(InputStream inputStream) {
         }
       }
     
    +  private static class WritableByteChannelImpl implements WritableByteChannel
    +  {
    +    private static final int TRANSFER_SIZE = 32 * 1024;
    +
    +    private OutputStream out;
    +    private final byte buffer[] = new byte[TRANSFER_SIZE];
    --- End diff --
    
    Unfortunately, this is one of the problems that the current design was intended to solve.
As we can see in the hash agg code; multiple of these objects are active in memory at any
one time. But, no more than one is ever writing. By putting the buffer here, we create one
per (potentially thousands) of retained channels. But, by putting the buffer on the allocator,
we hold one buffer per operator.
    
    Being the guy who wrote the current allocator code, I realize it is clunky. But, it is
clunky because we favored memory frugality over elegance. (If there is a cleaner way to have
one buffer per thread, I'm all for it!)
    
    Isn't the purpose of this change to avoid memory buffers? Seems, for the above reasons,
we're moving backwards...


> Avoid memory copy from direct buffer to heap while spilling to local disk
> -------------------------------------------------------------------------
>
>                 Key: DRILL-6002
>                 URL: https://issues.apache.org/jira/browse/DRILL-6002
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Vlad Rozov
>            Assignee: Vlad Rozov
>
> When spilling to a local disk or to any file system that supports WritableByteChannel
it is preferable to avoid copy from off-heap to java heap as WritableByteChannel can work
directly with the off-heap memory.  



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message