drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-5011) External Sort Batch memory use depends on record width
Date Wed, 09 Nov 2016 01:11:58 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15649214#comment-15649214
] 

Paul Rogers edited comment on DRILL-5011 at 11/9/16 1:11 AM:
-------------------------------------------------------------

Proposed solution:

The best solution is to revise the "copier" to track actual memory use. But, copier is generated
code so doing so is a big project. Short-term alternative: use a better record width estimate.

ESB already maintains a running cumulative record count.

Add a new count that is the in-memory count (total record count - spilled record count).

The ESB allocator already maintains a counter of the total memory allocated to the ESB (which,
by definition, must be for the in-memory batches.)

Compute a better record width estimate as:

{{record width estimate = allocated memory / in-memory record count}}

In a simple experiment:

{code}
Original record width estimate: 324
Copier allocator after copy: 320000, Records: 809 (x10)
Copier allocator after copy: 160256, Records: 809 (x11)
Copier allocator after copy: 80384, Records: 809 (x11)
Copier allocator after copy: 64000, Records: 809 (x many)

Revised record width estimate: 114
Copier allocator after copy: 320000, Records: 2299 (x10)
Copier allocator after copy: 221696, Records: 2299 (x many)
{code}

That is, with the original estimate, the copier did not make full use of the allocated memory
and copied fewer records. With the revised estimate, the copier copied more records per batch
and made better use of the allocated memory.

In this case, the real records were smaller than the original estimate, wasting memory. In
a query with larger records, the above estimate will prevent over-allocation of memory.



was (Author: paul-rogers):
Proposed solution:

The best solution is to revise the "copier" to track actual memory use. But, copier is generated
code so doing so is a big project. Short-term alternative: use a better record width estimate.

ESB already maintains a running cumulative record count.

Add a new count that is the in-memory count (total record count - spilled record count).

The ESB allocator already maintains a counter of the total memory allocated to the ESB (which,
by definition, must be for the in-memory batches.)

Compute a better record width estimate as:

{{record width estimate = allocated memory / in-memory record count}}

In a simple experiment:

{code}
Original record width estimate: 324
Revised record width estimate: 114
{code}



> External Sort Batch memory use depends on record width
> ------------------------------------------------------
>
>                 Key: DRILL-5011
>                 URL: https://issues.apache.org/jira/browse/DRILL-5011
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.8.0
>            Reporter: Paul Rogers
>            Priority: Minor
>
> The ExternalSortBatch operator uses spill-to-disk to keep memory needs within a defined
limit. However, the "copier" (really, the merge operation) can use an amount of memory determined
not by the operator configuration, but by the width of each record.
> The copier memory limit appears to be set by the COPIER_BATCH_MEM_LIMIT value.
> However, the actual memory use is determined by the number of records that the copier
is asked to copy. That record comes from an estimate of row width based on the type of each
column. Note that the row width *is not* based on the actual data in each row. Varchar fields,
for example, are assumed to be 40 characters wide. If the sorter is asked to sort records
with Varchar fields of, say, 1000 characters, then the row width estimate will be a poor estimator
of actual width.
> Memory use is based on a
> {code}
> target record count = memory limit / estimate row width
> {code}
> Actual memory use is:
> {code}
> memory use = target row count * actual row width
> {code}
> Which is
> {code}
> memory use = memory limit * actual row width  / estimate row width
> {code}
> That is, memory use depends on the ratio of actual to estimated width. If the estimate
is off by 2, then we use twice as much memory as expected.
> Not that the memory used for the copier defaults to 20 MB, so even an error of 4x still
means only 80 MB of memory used; small in comparison to the many GB typically allocated to
ESB storage.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message