drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5808) Reduce memory allocator strictness for "managed" operators
Date Tue, 26 Sep 2017 01:29:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16180052#comment-16180052
] 

ASF GitHub Bot commented on DRILL-5808:
---------------------------------------

Github user Ben-Zvi commented on a diff in the pull request:

    https://github.com/apache/drill/pull/958#discussion_r140870691
  
    --- Diff: exec/memory/base/src/main/java/org/apache/drill/exec/memory/Accountant.java
---
    @@ -22,16 +22,55 @@
     import javax.annotation.concurrent.ThreadSafe;
     
     import org.apache.drill.exec.exception.OutOfMemoryException;
    +import org.apache.drill.exec.util.AssertionUtil;
     
    +import com.google.common.annotations.VisibleForTesting;
     import com.google.common.base.Preconditions;
     
     /**
      * Provides a concurrent way to manage account for memory usage without locking. Used
as basis for Allocators. All
      * operations are threadsafe (except for close).
      */
     @ThreadSafe
    -class Accountant implements AutoCloseable {
    -  // private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(Accountant.class);
    +@VisibleForTesting
    +public class Accountant implements AutoCloseable {
    +  private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(Accountant.class);
    +
    +  // See DRILL-5808
    +  // Allow a "grace margin" above the allocator limit for those operators
    +  // that are trying to stay within the limit, but are hindered by the
    +  // current power-of-two allocations and random vector doublings. Instead
    +  // of failing the query, the accountant allows an excess allocation within
    +  // the grace, but issues a warning to the log.
    +  //
    +  // Grace is allowed only when requested, and then only in production
    +  // code, or if enabled in debug code by setting the following system
    +  // property.
    +
    +  public static final String STRICT_ALLOCATOR = "drill.memory.strict.allocator";
    +
    +  // Allow clients to allocate beyond the limit by this factor.
    +  // If the limit is 10K, then with a margin of 1, the system
    +  // will allow a grace of another 10K for a hard limit of 20K.
    +  // Warnings are issued for allocations that use the grace.
    +
    +  public static final int GRACE_MARGIN = 1;
    +
    +  // In large-memory allocators, don't allow more than 100 MB
    +  // grace.
    +
    +  public static final int MAX_GRACE = 100 * 1024 * 1024;
    +
    +  // Use the strictness set in a system property. If not set, then
    +  // default to strict in debug mode, lenient in production mode.
    +
    +  public static final boolean STRICT = System.getProperty(STRICT_ALLOCATOR) == null
    --- End diff --
    
    The name (in capitals) is a little confusing (e.g., when the value is false). So either
need to express that this is the value coming from the configuration, or something in low
(camel) case. 


> Reduce memory allocator strictness for "managed" operators
> ----------------------------------------------------------
>
>                 Key: DRILL-5808
>                 URL: https://issues.apache.org/jira/browse/DRILL-5808
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.11.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>             Fix For: 1.12.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Drill 1.11 and 1.12 introduce new "managed" versions of the sort and hash agg that enforce
memory limits, spilling to disk when necessary.
> Drill's internal memory system is very "lumpy" and unpredictable. The operators have
no control over the incoming batch size; an overly large batch can cause the operator to exceed
its memory limit before it has a chance to do any work.
> Vector allocations grow in power-of-two sizes. Adding a single record can double the
memory allocated to a vector.
> Drill has no metadata, so operators cannot predict the size of VarChar columns nor the
cardinality of arrays. The "Record Batch Sizer" tries to extract this information on each
batch, but it works with averages, and specific column patterns can still throw off the memory
calculations. (For example, having a series of very wide columns for A-M and very narrow columns
for N-Z will cause a moderate average. But, once sorted, the A-M rows, and batches, will be
much larger than expected, causing out-of-memory errors.)
> At present, if an operator is wrong in its memory usage by a single byte, the entire
query is killed. That is, the user pays the death penalty (of queries) for poor design decisions
within Drill. This leads to a less-than-optimal user experience.
> The proposal here is to make the memory allocator less strict for "managed" operators.
> First, we recognize that the managed operators do attempt to control memory and, if designed
well, will, on average hit their targets.
> Second, we recognize that, due to the lumpiness issues above, any single operator may
exceed, or be under, the configured maximum memory.
> Given this, the proposal here is:
> 1. An operator identifies itself as managed to the memory allocator.
> 2. In managed mode, the allocator has soft limits. It emits a warning to the log when
the limit is exceeded.
> 3. For safety, in managed mode, the allocator enforces a hard limit larger than the configured
limit.
> The enforcement limit might be:
> * For memory sizes < 100MB, up to 2x the configured limit.
> * For larger memory sizes, no more than 100MB over the configured limit.
> The exact numbers can be made configurable.
> Now, during testing, scripts should look for over-memory warnings. Each should be fixed
as we fix OOM issues today. But, during production, user queries are far less likely to fail
due to any remaining corner cases that throw off the memory calculations.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message