drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5014) ExternalSortBatch cache size, spill count differs from config setting
Date Tue, 28 Mar 2017 20:15:42 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15945869#comment-15945869
] 

Paul Rogers commented on DRILL-5014:
------------------------------------

Primarily a development issue; hard to test at the QA level.

> ExternalSortBatch cache size, spill count differs from config setting
> ---------------------------------------------------------------------
>
>                 Key: DRILL-5014
>                 URL: https://issues.apache.org/jira/browse/DRILL-5014
>             Project: Apache Drill
>          Issue Type: Sub-task
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>             Fix For: 1.11.0
>
>
> The ExternalSortBatch (ESB) operator sorts data while spilling to disk to remain within
a defined memory footprint. Spilling occurs based on a number of factors. Among those are
two config parameters:
> * {{drill.exec.sort.external.spill.group.size}}: The number of batches to spill per spill
event.
> * {{drill.exec.sort.external.spill.threshold}}: The number of batches to accumulate in
memory before starting a spill event.
> The expected behavior would be:
> * After the accumulated batches exceeds the threshold, and
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> That is if the threshold is 200, and the size is 150, we should accumulate 200 batches,
then spill 150 of them (leaving 50) and repeat.
> The actual behavior is:
> * When the accumulated records exceeds the threshold and,
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> The above can leave more batches in memory than desired, and spill more than desired.
> The actual behavior for the (threshold=200, size=150) case is:
> {code}
> Before spilling, buffered batch count: 201
> After spilling, buffered batch count: 101
> Before spilling, buffered batch count: 251
> After spilling, buffered batch count: 126
> Before spilling, buffered batch count: 276
> After spilling, buffered batch count: 138
> Before spilling, buffered batch count: 288
> After spilling, buffered batch count: 144
> Before spilling, buffered batch count: 294
> After spilling, buffered batch count: 147
> Before spilling, buffered batch count: 297
> After spilling, buffered batch count: 149
> Before spilling, buffered batch count: 299
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> {code}
> In short, the actual number of batches retained in memory is twice the spill size, **NOT**
the number set in the threshold. As a result, the ESB operator will use more memory than expected.
> The work-around is to set a batch size that is half the threshold so that the batch size
(used in spill decisions) matches the actual spill count (as implemented by the code.)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message