drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From paul-rogers <...@git.apache.org>
Subject [GitHub] drill issue #1075: DRILL-6030: Managed sort should minimize number of batche...
Date Tue, 19 Dec 2017 18:39:03 GMT
Github user paul-rogers commented on the issue:

    https://github.com/apache/drill/pull/1075
  
    One additional thought. This bug was found when sorting 18 GB of data in 8 GB of memory.
That is, a case in which the sort must spill.
    
    What happens in the case in which the 18 GB of data is sorted in, say, 20 GB of memory
(an in-memory sort)? We don't want the merge limit to force a spill in this case; kind of
defeats the purpose of an in-memory sort.
    
    So:
    
    1. Does the limit affect in memory sort? If so, we need to revise the solution.
    2. Does the in-memory sort suffer from a similar performance issue? If so, we need to
revise the in memory sort.
    
    One possible solution is to:
    
    1. Defer sorting of individual batches until necessary.
    2. Sort batches just before spilling.
    3. If all batches fit in memory, do a single, combined sort (using an SV4).


---

Mime
View raw message