drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-5014) ExternalSortBatch cache size, spill count differs from config setting
Date Tue, 08 Nov 2016 23:57:59 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15648544#comment-15648544
] 

Paul Rogers edited comment on DRILL-5014 at 11/8/16 11:57 PM:
--------------------------------------------------------------

The issue actually seems more complex. ESB seems to have evolved to have several different
ways to spill that all interact in complex ways.

On the one hand. ESB tries to operate within a defined memory "budget." In this model, ESB
accumulates batches until the memory used exceeds 95% of the allowed maximum. Then, ESB spills
half the batches. Simple enough.

The maximum memory comes from two sources: either a limit set by the Foreman or a limit set
by the "parent allocator" in the operator tree. (One part of the code uses one limit, another
part uses the other limit.)

Then, the two config vars above were added. They create different ideas about limits and when
to spill, causing the unexpected behavior described above.

But, the actual situation is even more complex. The code started by spilling only half the
in-memory batches. Makes sense: don't spill more than needed. But, the implementation of {{mergeSpilled()}}
immediately spills all remaining in-memory batches to disk so it can use the disk-based merge
utility rather than a hybrid in-memory/on-disk tool (which does not exist.) Given this, there
is no reason to hold onto any batches in memory once spilling starts: they'll all be written
to disk later anyway.

What is needed is a rationalized design. Is it the memory limit or spill counts that determine
spilling? What if spill counts conflict with memory? If memory-based spilling conflicts with
the spill parameters?


was (Author: paul-rogers):
The issue actually seems more complex. ESB seems to have evolved to have several different
ways to spill that all interact in complex ways.

On the one hand. ESB tries to operate within a defined memory "budget." In this model, ESB
accumulates batches until the memory used exceeds 95% of the allowed maximum. Then, ESB spills
half the batches. Simple enough.

The maximum memory comes from two sources: either a limit set by the Foreman or a limit set
by the "parent allocator" in the operator tree. (One part of the code uses one limit, another
part uses the other limit.)

Then, the two config vars above were added. They create different ideas about limits and when
to spill, causing the unexpected behavior described above.

What is needed is a rationalized design. Is it the memory limit or spill counts that determine
spilling? What if spill counts conflict with memory? If memory-based spilling conflicts with
the spill parameters?

> ExternalSortBatch cache size, spill count differs from config setting
> ---------------------------------------------------------------------
>
>                 Key: DRILL-5014
>                 URL: https://issues.apache.org/jira/browse/DRILL-5014
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Priority: Minor
>
> The ExternalSortBatch (ESB) operator sorts data while spilling to disk to remain within
a defined memory footprint. Spilling occurs based on a number of factors. Among those are
two config parameters:
> * {{drill.exec.sort.external.spill.group.size}}: The number of batches to spill per spill
event.
> * {{drill.exec.sort.external.spill.threshold}}: The number of batches to accumulate in
memory before starting a spill event.
> The expected behavior would be:
> * After the accumulated batches exceeds the threshold, and
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> That is if the threshold is 200, and the size is 150, we should accumulate 200 batches,
then spill 150 of them (leaving 50) and repeat.
> The actual behavior is:
> * When the accumulated records exceeds the threshold and,
> * More than "batch size" new batches have arrived since the last spill,
> * Spill half the accumulated records.
> The above can leave more batches in memory than desired, and spill more than desired.
> The actual behavior for the (threshold=200, size=150) case is:
> {code}
> Before spilling, buffered batch count: 201
> After spilling, buffered batch count: 101
> Before spilling, buffered batch count: 251
> After spilling, buffered batch count: 126
> Before spilling, buffered batch count: 276
> After spilling, buffered batch count: 138
> Before spilling, buffered batch count: 288
> After spilling, buffered batch count: 144
> Before spilling, buffered batch count: 294
> After spilling, buffered batch count: 147
> Before spilling, buffered batch count: 297
> After spilling, buffered batch count: 149
> Before spilling, buffered batch count: 299
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> After spilling, buffered batch count: 150
> Before spilling, buffered batch count: 300
> {code}
> In short, the actual number of batches retained in memory is twice the spill size, **NOT**
the number set in the threshold. As a result, the ESB operator will use more memory than expected.
> The work-around is to set a batch size that is half the threshold so that the batch size
(used in spill decisions) matches the actual spill count (as implemented by the code.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message