Mailing-List: contact dev-help@drill.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@drill.apache.org
From: paul-rogers <git@git.apache.org>
To: dev@drill.apache.org
Reply-To: dev@drill.apache.org
References: <git-pr-928-drill@git.apache.org>
In-Reply-To: <git-pr-928-drill@git.apache.org>
Subject: [GitHub] drill pull request #928: DRILL-5716: Queue-driven memory allocation
Content-Type: text/plain
Message-Id: <20170901215103.20DC7F5658@git1-us-west.apache.org>
Date: Fri,  1 Sep 2017 21:51:03 +0000 (UTC)
archived-at: Fri, 01 Sep 2017 21:51:09 -0000

Github user paul-rogers commented on a diff in the pull request:

    https://github.com/apache/drill/pull/928#discussion_r136671513
  
    --- Diff: exec/java-exec/src/main/resources/drill-module.conf ---
    @@ -344,6 +357,20 @@ drill.exec: {
       # Full workspace name should be indicated (including schema and workspace separated by dot).
       # Workspace MUST be file-based and writable. Workspace name is case-sensitive.
       default_temporary_workspace: "dfs.tmp"
    +
    +  // Resource management
    +  rm : {
    +    // Memory per node normally comes from the direct memory alloated on the JVM
    +    // command line. This parameter, if other than 0, further limits the amount.
    +    // Primarily for testing.
    +    memory_per_node: 0,
    +    // The number of available CPUs normally comes directly from the system itself.
    +    // This parameter, if other than 0, further limits the number of CPUs will
    +    // will consider when planning. Note that, sadly, this parameter does not
    +    // limit *actual* CPU usage; only the amount of CPU assumed to exist when
    +    // planning and managing queries. Primarily for testing.
    +    cpus_per_node: 0,
    +  }
    --- End diff --
    
    Well, it is not for testing only. Suppose that Drill runs under YARN. Then Drill should not use the full set of CPUs; it should use the number of vcores granted by YARN. In this case, the YARN launcher can use `-Ddrill.exec.rm.cpus_per_node=5` to specify the number of vcores.
    
    The memory per node is a safety-valve. At present, several operators (at least three) decline to participate in the memory assignment scheme. We may find it necessary to artificially limit memory in order to force certain queries to work if they end up using far more memory than expected. (True, the memory ratio runtime option also solves this issue...)
    
    These options do not appear in `drill-override-example.conf`, so users won't know about them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---