drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From paul-rogers <...@git.apache.org>
Subject [GitHub] drill pull request #958: DRILL-5808: Reduce memory allocator strictness for ...
Date Tue, 26 Sep 2017 01:35:24 GMT
Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/958#discussion_r140939480
  
    --- Diff: exec/memory/base/src/main/java/org/apache/drill/exec/memory/Accountant.java
---
    @@ -22,16 +22,55 @@
     import javax.annotation.concurrent.ThreadSafe;
     
     import org.apache.drill.exec.exception.OutOfMemoryException;
    +import org.apache.drill.exec.util.AssertionUtil;
     
    +import com.google.common.annotations.VisibleForTesting;
     import com.google.common.base.Preconditions;
     
     /**
      * Provides a concurrent way to manage account for memory usage without locking. Used
as basis for Allocators. All
      * operations are threadsafe (except for close).
      */
     @ThreadSafe
    -class Accountant implements AutoCloseable {
    -  // private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(Accountant.class);
    +@VisibleForTesting
    +public class Accountant implements AutoCloseable {
    +  private static final org.slf4j.Logger logger = org.slf4j.LoggerFactory.getLogger(Accountant.class);
    +
    +  // See DRILL-5808
    +  // Allow a "grace margin" above the allocator limit for those operators
    +  // that are trying to stay within the limit, but are hindered by the
    +  // current power-of-two allocations and random vector doublings. Instead
    +  // of failing the query, the accountant allows an excess allocation within
    +  // the grace, but issues a warning to the log.
    +  //
    +  // Grace is allowed only when requested, and then only in production
    +  // code, or if enabled in debug code by setting the following system
    +  // property.
    +
    +  public static final String STRICT_ALLOCATOR = "drill.memory.strict.allocator";
    +
    +  // Allow clients to allocate beyond the limit by this factor.
    +  // If the limit is 10K, then with a margin of 1, the system
    +  // will allow a grace of another 10K for a hard limit of 20K.
    +  // Warnings are issued for allocations that use the grace.
    +
    +  public static final int GRACE_MARGIN = 1;
    +
    +  // In large-memory allocators, don't allow more than 100 MB
    +  // grace.
    +
    +  public static final int MAX_GRACE = 100 * 1024 * 1024;
    +
    +  // Use the strictness set in a system property. If not set, then
    +  // default to strict in debug mode, lenient in production mode.
    +
    +  public static final boolean STRICT = System.getProperty(STRICT_ALLOCATOR) == null
    --- End diff --
    
    This is a constant, hence the caps. It may be a bit confusing since it is set as an expression,
but that is perfectly valid Java.


---

Mime
View raw message