drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5284) Roll-up of final fixes for managed sort
Date Tue, 28 Feb 2017 00:06:46 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886810#comment-15886810
] 

ASF GitHub Bot commented on DRILL-5284:
---------------------------------------

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/761#discussion_r103332364
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/managed/ExternalSortBatch.java
---
    @@ -392,22 +448,31 @@ private void configure(DrillConfig config) {
         // Set too large and the ratio between memory and input data sizes becomes
         // small. Set too small and disk seek times dominate performance.
     
    -    spillBatchSize = config.getBytes(ExecConstants.EXTERNAL_SORT_SPILL_BATCH_SIZE);
    -    spillBatchSize = Math.max(spillBatchSize, MIN_SPILL_BATCH_SIZE);
    +    preferredSpillBatchSize = config.getBytes(ExecConstants.EXTERNAL_SORT_SPILL_BATCH_SIZE);
    +
    +    // In low memory, use no more than 1/4 of memory for each spill batch. Ensures we
    +    // can merge.
    +
    +    preferredSpillBatchSize = Math.min(preferredSpillBatchSize, memoryLimit / 4);
    --- End diff --
    
    In low memory conditions, restrict the spill batch size to 1/4 of memory. Why?
    
    * We need to accumulate at least 2 such batches to do a merge. (Now at 1/2 of memory.)
    * We need to create an output batch from the two inputs (3/4 of memory).
    * Need overhead for other direct memory uses. (Remaining 1/4 of memory.)
    
    Sadly, memory management in Drill is not very precise: batch sizes can't be predicted
with any accuracy. Trying to use, say, 1/3 of memory for the spill batch would seem more logical.
(Two batches into the merge, one out), but the allocator issues a fatal error if we guess
wrong by even one byte. So, we are forced to be conservative.
    
    If we had better control, or a more forgiving allocator, we could make different choices.
    
    Also, why try to sort GBs of data in 20 MB? Yet, this is the test case that had to be
solved and that this particular fix enables.
    
    I'm open to suggestions for better solutions; this is a very tricky area...


> Roll-up of final fixes for managed sort
> ---------------------------------------
>
>                 Key: DRILL-5284
>                 URL: https://issues.apache.org/jira/browse/DRILL-5284
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.10.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>             Fix For: 1.10.0
>
>
> The managed external sort was introduced in DRILL-5080. Since that time, extensive testing
has identified a number of minor fixes and improvements. Given the long PR cycles, it is not
practical to spend a week or two to do a PR for each fix individually. This ticket represents
a roll-up of a combination of a number of fixes. Small fixes are listed here, larger items
appear as sub-tasks.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message