drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kunal Khatua (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-5741) Drillbit during startup should not exceed the available memory on a node
Date Fri, 25 Aug 2017 20:08:00 GMT
Kunal Khatua created DRILL-5741:
-----------------------------------

             Summary: Drillbit during startup should not exceed the available memory on a
node
                 Key: DRILL-5741
                 URL: https://issues.apache.org/jira/browse/DRILL-5741
             Project: Apache Drill
          Issue Type: Improvement
          Components:  Server
    Affects Versions: 1.11.0
            Reporter: Kunal Khatua
             Fix For: 1.12.0


Currently, during startup, a Drillbit can be assigned large values for the following:
* Xmx (Heap)
* XX:MaxDirectMemorySize
* XX:ReservedCodeCacheSize
* XX:MaxPermSize

All of this, potentially, can exceed the available memory on a system when a Drillbit is under
heavy load. It would be good to have the Drillbit ensure during startup itself that the cumulative
value of these parameters does not exceed a pre-defined upper limit for the Drill process.

The proposal is to have the [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
script look for an additional environment variable:
{{DRILLBIT_MAX_PROC_MEM}}

The parameter can specify the maximum in GB/MB (similar in syntax to how the Java's MaxHeap
is defined), or in terms of percentage of available memory (not to exceed 95%).

The [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
script will perform the calculation of the sum of memory required by the memory spaces (heap,
direct, etc) and ensure that it is within the limit defined by the {{DRILLBIT_MAX_PROC_MEM}}
env variable. 

In the absence of this parameter, there will be no restriction. A node admin can then define
this variable in the default terminal's environment (e.g. {{/root/.bashrc}} ) files.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message