drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5741) Automatically manage memory allocations during startup
Date Wed, 10 Jan 2018 21:30:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321142#comment-16321142

ASF GitHub Bot commented on DRILL-5741:

Github user kkhatua commented on a diff in the pull request:

    --- Diff: distribution/src/resources/drill-config.sh ---
    @@ -180,18 +251,46 @@ else
    -# Default memory settings if none provided by the environment or
    +# Checking if being executed in context of Drillbit and not SQLLine
    +if [ "$DRILLBIT_CONTEXT" == "1" ]; then 
    +  # *-auto.sh allows for distrib/user specific checks to be done
    +  distribAuto="$DRILL_CONF_DIR/distrib-auto.sh"
    +  if [ ! -r "$distribAuto" ]; then distribAuto="$DRILL_HOME/conf/distrib-auto.sh"; fi
    +  if [ ! -r "$distribAuto" ]; then distribAuto=""; fi
    +  drillAuto="$DRILL_CONF_DIR/drill-auto.sh"
    +  if [ ! -r "$drillAuto" ]; then drillAuto="$DRILL_HOME/conf/drill-auto.sh"; fi
    +  if [ ! -r "$drillAuto" ]; then drillAuto=""; fi
    +  # Enforcing checks in order (distrib-auto.sh , drill-auto.sh)
    +  # (NOTE: A script is executed only if it has relevant executable lines)
    +  if [ -n "$distribAuto" ] && [ $(executableLineCount $distribAuto) -gt 0 ];
    +    . "$distribAuto"
    +    if [ $? -gt 0 ]; then fatal_error "Aborting Drill Startup due failed checks from
$distribAuto"; fi
    +  fi
    +  if [ -n "$drillAuto" ] && [ $(executableLineCount $drillAuto) -gt 0 ]; then
    --- End diff --
    Passed the checks for the file to the renamed function: `checkExecutableLineCount`

> Automatically manage memory allocations during startup
> ------------------------------------------------------
>                 Key: DRILL-5741
>                 URL: https://issues.apache.org/jira/browse/DRILL-5741
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components:  Server
>    Affects Versions: 1.11.0
>            Reporter: Kunal Khatua
>            Assignee: Kunal Khatua
>             Fix For: 1.13.0
>         Attachments: Auto Mem Allocation Proposal - Computation Logic.pdf, Auto Mem Allocation
Proposal - Scenarios.pdf
> Currently, during startup, a Drillbit can be assigned large values for the following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a Drillbit
is under heavy load. It would be good to have the Drillbit ensure during startup itself that
the cumulative value of these parameters does not exceed a pre-defined upper limit for the
Drill process.
> This JIRA is a *proposal* to allow for automatic configuration (based on configuration
patterns observed in production Drill clusters). It leverages the capability of providing
distribution (and user-specific) checks during Drill Startup from DRILL-6068.
> The idea is to remove the need for a user to worry about managing the tuning parameters,
by providing the optimal values. In addition, it also allows for the memory allocation to
be implicitly managed by simply providing the Drill process with a single dimensional of total
process memory (either in absolute values, or as a percentage of the total system memory),
while {{distrib-auto.sh}} provides the individual allocations.
> This allocation is then partitioned into allocations for Heap and Direct Memory, with
a small portion allocated for the Generated Java CodeCache as well. If any of the individual
allocations are also specified (via {{distrib-env.sh}} or {{drill-env.sh}}), the remaining
unspecified allocations are adjusted to stay +within the limits+ of the total memory allocation.
> The *details* of the proposal are here:
> https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0
> For those unable to access the Google Document, PDFs are attached:
> * [^Auto Mem Allocation Proposal - Computation Logic.pdf] - Provides the equation used
for computing the heap, direct and code cache allocations for a given input
> * [^Auto Mem Allocation Proposal - Scenarios.pdf] - Describes the various inputs, and
their expected allocations
> The variables that are (_optionally_) defined (in memory, {{distrib-env.sh}} or {{drill-env.sh}}
) are:
> * {{DRILLBIT_MAX_PROC_MEM}} : Total Process Memory
> * {{DRILL_HEAP}} : JVM Max Heap Size
> * {{DRILL_MAX_DIRECT_MEMORY}} : JVM Max Direct Memory Size
> * {{DRILLBIT_CODE_CACHE_SIZE}} : JVM Code Cache Size
> Note: _With JDK8, MaxPermSize is no longer supported, so we do not account for this any
more, and will unset the variable if JDK8 or higher is detected._

This message was sent by Atlassian JIRA

View raw message