From issues-return-49221-archive-asf-public=cust-asf.ponee.io@drill.apache.org Mon Jan 29 04:06:31 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id 40052180799 for ; Mon, 29 Jan 2018 04:06:31 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 30328160C43; Mon, 29 Jan 2018 03:06:31 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7E302160C55 for ; Mon, 29 Jan 2018 04:06:30 +0100 (CET) Received: (qmail 85442 invoked by uid 500); 29 Jan 2018 03:06:18 -0000 Mailing-List: contact issues-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@drill.apache.org Delivered-To: mailing list issues@drill.apache.org Received: (qmail 84761 invoked by uid 99); 29 Jan 2018 03:06:15 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jan 2018 03:06:15 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 06AC81A231A for ; Mon, 29 Jan 2018 03:06:15 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -102.311 X-Spam-Level: X-Spam-Status: No, score=-102.311 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id he2Ft9Fif089 for ; Mon, 29 Jan 2018 03:06:10 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id C633160F20 for ; Mon, 29 Jan 2018 03:06:09 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id B7393E019C for ; Mon, 29 Jan 2018 03:06:07 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 1B0B9240F6 for ; Mon, 29 Jan 2018 03:06:04 +0000 (UTC) Date: Mon, 29 Jan 2018 03:06:04 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: issues@drill.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (DRILL-5741) Automatically manage memory allocations during startup MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/DRILL-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342839#comment-16342839 ] ASF GitHub Bot commented on DRILL-5741: --------------------------------------- Github user paul-rogers commented on a diff in the pull request: https://github.com/apache/drill/pull/1082#discussion_r164324543 --- Diff: distribution/src/assemble/bin.xml --- @@ -345,6 +345,21 @@ 0755 conf + + src/resources/auto-setup.sh + 0755 + bin + + + src/resources/drill-setup.sh + 0755 + conf + + + src/resources/distrib-setup.sh --- End diff -- There should be no "distrib" files in Drill. The scripts should check if they exist, and if not (the default), just ignore the file. Distributions add their own files. Having no default file makes clear that these are not Apache-provided files. > Automatically manage memory allocations during startup > ------------------------------------------------------ > > Key: DRILL-5741 > URL: https://issues.apache.org/jira/browse/DRILL-5741 > Project: Apache Drill > Issue Type: Improvement > Components: Server > Affects Versions: 1.11.0 > Reporter: Kunal Khatua > Assignee: Kunal Khatua > Priority: Major > Fix For: 1.13.0 > > Attachments: Auto Mem Allocation Proposal - Computation Logic.pdf, Auto Mem Allocation Proposal - Scenarios.pdf > > > Currently, during startup, a Drillbit can be assigned large values for the following: > * Xmx (Heap) > * XX:MaxDirectMemorySize > * XX:ReservedCodeCacheSize > * XX:MaxPermSize > All of this, potentially, can exceed the available memory on a system when a Drillbit is under heavy load. It would be good to have the Drillbit ensure during startup itself that the cumulative value of these parameters does not exceed a pre-defined upper limit for the Drill process. > This JIRA is a *proposal* to allow for automatic configuration (based on configuration patterns observed in production Drill clusters). It leverages the capability of providing default/distribution (and user-specific) checks during Drill Startup from DRILL-6068. > The idea is to remove the need for a user to worry about managing the tuning parameters, by providing the optimal values. In addition, it also allows for the memory allocation to be implicitly managed by simply providing the Drill process with a single dimensional of total process memory (either in absolute values, or as a percentage of the total system memory), while {{auto-setup.sh}} provides the individual allocations. > This allocation is then partitioned into allocations for Heap and Direct Memory, with a small portion allocated for the Generated Java CodeCache as well. If any of the individual allocations are also specified (via {{distrib-env.sh}} or {{drill-env.sh}}), the remaining unspecified allocations are adjusted to stay +within the limits+ of the total memory allocation. > The *details* of the proposal are here: > https://docs.google.com/spreadsheets/d/1N6VYlQFiPoTV4iD46XbkIrvEQesiGFUU9-GWXYsAPXs/edit#gid=0 > For those unable to access the Google Document, PDFs are attached: > * [^Auto Mem Allocation Proposal - Computation Logic.pdf] - Provides the equation used for computing the heap, direct and code cache allocations for a given input > * [^Auto Mem Allocation Proposal - Scenarios.pdf] - Describes the various inputs, and their expected allocations > The variables that are (_optionally_) defined (in memory, {{distrib-env.sh}} or {{drill-env.sh}} ) are: > * {{DRILLBIT_MAX_PROC_MEM}} : Total Process Memory > * {{DRILL_HEAP}} : JVM Max Heap Size > * {{DRILL_MAX_DIRECT_MEMORY}} : JVM Max Direct Memory Size > * {{DRILLBIT_CODE_CACHE_SIZE}} : JVM Code Cache Size > Note: _With JDK8, MaxPermSize is no longer supported, so we do not account for this any more, and will unset the variable if JDK8 or higher is detected._ -- This message was sent by Atlassian JIRA (v7.6.3#76005)