Mailing-List: contact issues-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
Date: Fri, 25 Aug 2017 20:48:00 +0000 (UTC)
From: "Paul Rogers (JIRA)" <jira@apache.org>
To: issues@drill.apache.org
Message-ID: <JIRA.13097695.1503691625000.123762.1503694080468@Atlassian.JIRA>
In-Reply-To: <JIRA.13097695.1503691625000@Atlassian.JIRA>
References: <JIRA.13097695.1503691625000@Atlassian.JIRA> <JIRA.13097695.1503691625954@jira-lw-us.apache.org>
Subject: [jira] [Commented] (DRILL-5741) Drillbit during startup should not
 exceed the available memory on a node
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 25 Aug 2017 20:48:06 -0000


    [ https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142160#comment-16142160 ] 

Paul Rogers commented on DRILL-5741:
------------------------------------

Not sure this entirely makes sense. It is again asking the user to add a new variable to check the user's other settings.

In general, Drill should not use the entire memory on a node.

When running under YARN, YARN will assign memory. When running under other managers (MapR Warden, Mesos, etc.) then those systems take care of the total memory allocations across tasks.

Perhaps we could, on Drillbit start, sum the memory allocations and check against total OS memory. But, how much should we reserve for the OS? For file system caching? For ZK? For other apps? Pretty soon we are trying to do node-level resource management "blind" inside the Drillbit.

In Drill-on-YARN, we considered the percentage-based allocation suggested above. But, this is not as simple as it seems. Certain memory units are fixed (such as code cache), some can be adjusted. But should the ratio between heap and direct be the same at small levels (2 GB and 4 GB, say) vs at large levels (50 GB and 100 GB?).

Instead, we worked the other way. We summed the memory allocation for code cache, heap and direct to get the total memory requested from YARN.

I think we can consider memory oversubscription as a user error; a bit like configuring storage plugins wrong, or running too many processes for a node, or configuring the OS wrong, etc.

> Drillbit during startup should not exceed the available memory on a node
> ------------------------------------------------------------------------
>
>                 Key: DRILL-5741
>                 URL: https://issues.apache.org/jira/browse/DRILL-5741
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components:  Server
>    Affects Versions: 1.11.0
>            Reporter: Kunal Khatua
>             Fix For: 1.12.0
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> Currently, during startup, a Drillbit can be assigned large values for the following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a Drillbit is under heavy load. It would be good to have the Drillbit ensure during startup itself that the cumulative value of these parameters does not exceed a pre-defined upper limit for the Drill process.
> The proposal is to have the [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit] script look for an additional environment variable:
> {{DRILLBIT_MAX_PROC_MEM}}
> The parameter can specify the maximum in GB/MB (similar in syntax to how the Java's MaxHeap is defined), or in terms of percentage of available memory (not to exceed 95%).
> The [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit] script will perform the calculation of the sum of memory required by the memory spaces (heap, direct, etc) and ensure that it is within the limit defined by the {{DRILLBIT_MAX_PROC_MEM}} env variable. 
> In the absence of this parameter, there will be no restriction. A node admin can then define this variable in the default terminal's environment (e.g. {{/root/.bashrc}} ) files.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)