drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5019) ExternalSortBatch spills all batches to disk even if even one spills
Date Tue, 28 Mar 2017 20:15:42 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945868#comment-15945868

Paul Rogers commented on DRILL-5019:

Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch spills all batches to disk even if even one spills
> --------------------------------------------------------------------
>                 Key: DRILL-5019
>                 URL: https://issues.apache.org/jira/browse/DRILL-5019
>             Project: Apache Drill
>          Issue Type: Sub-task
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>             Fix For: 1.11.0
> The ExternalSortBatch (ESB) operator sorts batches while spilling to disk to stay within
a defined memory budget.
> Assume the memory budget is 10 GB. Assume that the actual volume of data to be sorted
is 10.1 GB. The ESB spills the extra 0.1 GB to disk. (Actually spills more than that, say
5 GB.)
> At the completion of the run, ESB has read all incoming batches. It must now merge those
batches. It does so by spilling **all** batches to disk, then doing a disk-based merge.
> This means that exceeding the memory limit by even a small amount is the same as having
a very low memory limit: all batches must spill.
> This solution is simple, it works, and has some amount of logic.
> But, it would be better to have a slightly more advanced solution that spills only the
smallest possible set of batches to disk, then does a hybrid in-memory, on-disk merge, saving
the unnecessary write/read cycle.

This message was sent by Atlassian JIRA

View raw message