drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5275) Sort spill serialization is slow due to repeated buffer allocations
Date Wed, 22 Feb 2017 03:51:44 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877425#comment-15877425
] 

ASF GitHub Bot commented on DRILL-5275:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/754#discussion_r102378905
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/cache/VectorAccessibleSerializable.java
---
    @@ -57,6 +57,12 @@
       private BatchSchema.SelectionVectorMode svMode = BatchSchema.SelectionVectorMode.NONE;
       private SelectionVector2 sv2;
     
    +  /**
    +   * Disk I/O buffer used for all reads and writes of DrillBufs.
    +   */
    +
    +  private byte buffer[] = new byte[32*1024];
    --- End diff --
    
    Read a 18 GB file on disk using just a test program that uses various size buffers.
    
    {code}
    32K buffer: Rate: 799 MB/s
    64K buffer: Rate: 766 MB/s
    {code}
    
    So, seems no advantage of a larger buffer. (Tests with a smaller buffer do slow things
down, hence the 32K size.)
    
    On direct memory: can't use direct memory as the fundamental problem is that data is in
a direct memory DrillBuf, and must be copied to heap memory for writing. The original code
does the copy by allocating a heap buffer the same size as the vector (16 MB, 32 MB or larger.)
This code does the copying by reusing the same buffer over and over.
    
    No need to hold the buffer on the operator. This class is used for an entire spill/read
session.
    
    What may be an issue, however, is the merge phase of a sort when many files are open and
so many buffers are created. The reads are synchronous, so they could share a buffer.


> Sort spill serialization is slow due to repeated buffer allocations
> -------------------------------------------------------------------
>
>                 Key: DRILL-5275
>                 URL: https://issues.apache.org/jira/browse/DRILL-5275
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.10.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>             Fix For: 1.10.0
>
>
> Drill provides a sort operator that spills to disk. The spill and read operations use
the serialization code in the {{VectorAccessibleSerializable}}. This code, in turn, uses the
{{DrillBuf.getBytes()}} method to write to an output stream. (Yes, the "get" method writes,
and the "write" method reads...)
> The DrillBuf method turns around and calls the UDLE method that does:
> {code}
>             byte[] tmp = new byte[length];
>             PlatformDependent.copyMemory(addr(index), tmp, 0, length);
>             out.write(tmp);
> {code}
> That is, for each write the code allocates a heap buffer. Since Drill buffers can be
quite large (4, 8, 16 MB or larger), the above rapidly fills the heap and causes GC.
> The result is slow performance. On a Mac, with an SSD that can do 700 MB/s of I/O, we
get only about 40 MB/s. Very likely because of excessive CPU cost and GC.
> The solution is to allocate a single read or write buffer, then use that same buffer
over and over when reading or writing. This must be done in {{VectorAccessibleSerializable}}
as it is a per-thread class that has visibility to all the buffers to be written.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message