drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5741) During startup Drill should not exceed the available memory
Date Mon, 28 Aug 2017 20:48:00 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144315#comment-16144315

ASF GitHub Bot commented on DRILL-5741:

Github user paul-rogers commented on the issue:

    @kkhatua, thanks for the note. No doubt the proposed model helps in _some_ cases. The
problem is, it adds complexity for other cases.
    Perhaps offer the change as an additional script that can be called from `drill_env.sh`
for those sites that need it, rather than making it part of the core, common script. That
way, this solution is orthogonal to, rather than part of, a solution driven by a resource
manager such as YARN, Mesos or Warden.

> During startup Drill should not exceed the available memory
> -----------------------------------------------------------
>                 Key: DRILL-5741
>                 URL: https://issues.apache.org/jira/browse/DRILL-5741
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components:  Server
>    Affects Versions: 1.11.0
>            Reporter: Kunal Khatua
>            Assignee: Kunal Khatua
>             Fix For: 1.12.0
>   Original Estimate: 48h
>  Remaining Estimate: 48h
> Currently, during startup, a Drillbit can be assigned large values for the following:
> * Xmx (Heap)
> * XX:MaxDirectMemorySize
> * XX:ReservedCodeCacheSize
> * XX:MaxPermSize
> All of this, potentially, can exceed the available memory on a system when a Drillbit
is under heavy load. It would be good to have the Drillbit ensure during startup itself that
the cumulative value of these parameters does not exceed a pre-defined upper limit for the
Drill process.
> The proposal is to have the [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
script look for an additional environment variable:
> The parameter can specify the maximum in GB/MB (similar in syntax to how the Java's MaxHeap
is defined), or in terms of percentage of available memory (not to exceed 95%).
> The [runbit|https://github.com/apache/drill/blob/master/distribution/src/resources/runbit]
script will perform the calculation of the sum of memory required by the memory spaces (heap,
direct, etc) and ensure that it is within the limit defined by the {{DRILLBIT_MAX_PROC_MEM}}
env variable. 
> In the absence of this parameter, there will be no restriction. A node admin can then
define this variable in the default terminal's environment (e.g. {{/root/.bashrc}} ) files.

This message was sent by Atlassian JIRA

View raw message