drill-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bridg...@apache.org
Subject drill git commit: NR's multitenant changes-partial
Date Thu, 07 May 2015 00:37:04 GMT
Repository: drill
Updated Branches:
  refs/heads/gh-pages 27067e8c5 -> bca47014e


NR's multitenant changes-partial


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/bca47014
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/bca47014
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/bca47014

Branch: refs/heads/gh-pages
Commit: bca47014e556ac41809c77aa1f715731318b73f9
Parents: 27067e8
Author: Kristine Hahn <khahn@maprtech.com>
Authored: Wed May 6 17:21:03 2015 -0700
Committer: Kristine Hahn <khahn@maprtech.com>
Committed: Wed May 6 17:21:03 2015 -0700

----------------------------------------------------------------------
 _data/docs.json                                 | 44 ++++++++++----------
 ...-configuring-drill-in-a-dedicated-cluster.md | 30 -------------
 .../020-configuring-drill-memory.md             | 37 ++++++++++++++++
 .../050-configuring-multitenant-resources.md    | 24 ++++++++++-
 4 files changed, 82 insertions(+), 53 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/drill/blob/bca47014/_data/docs.json
----------------------------------------------------------------------
diff --git a/_data/docs.json b/_data/docs.json
index 2da89e5..4aca3af 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -821,8 +821,8 @@
                         }
                     ], 
                     "children": [], 
-                    "next_title": "Configuring Drill in a Dedicated Cluster", 
-                    "next_url": "/docs/configuring-drill-in-a-dedicated-cluster/", 
+                    "next_title": "Configuring Drill Memory", 
+                    "next_url": "/docs/configuring-drill-memory/", 
                     "parent": "Configure Drill", 
                     "previous_title": "Configure Drill", 
                     "previous_url": "/docs/configure-drill/", 
@@ -843,9 +843,9 @@
                     "parent": "Configure Drill", 
                     "previous_title": "Configure Drill Introduction", 
                     "previous_url": "/docs/configure-drill-introduction/", 
-                    "relative_path": "_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md",

-                    "title": "Configuring Drill in a Dedicated Cluster", 
-                    "url": "/docs/configuring-drill-in-a-dedicated-cluster/"
+                    "relative_path": "_docs/configure-drill/020-configuring-drill-memory.md",

+                    "title": "Configuring Drill Memory", 
+                    "url": "/docs/configuring-drill-memory/"
                 }, 
                 {
                     "breadcrumbs": [
@@ -922,8 +922,8 @@
                     "next_title": "Configuring a Multitenant Cluster Introduction", 
                     "next_url": "/docs/configuring-a-multitenant-cluster-introduction/",

                     "parent": "Configure Drill", 
-                    "previous_title": "Configuring Drill in a Dedicated Cluster", 
-                    "previous_url": "/docs/configuring-drill-in-a-dedicated-cluster/", 
+                    "previous_title": "Configuring Drill Memory", 
+                    "previous_url": "/docs/configuring-drill-memory/", 
                     "relative_path": "_docs/configure-drill/040-configuring-a-multitenant-cluster.md",

                     "title": "Configuring a Multitenant Cluster", 
                     "url": "/docs/configuring-a-multitenant-cluster/"
@@ -1099,8 +1099,8 @@
                 }
             ], 
             "children": [], 
-            "next_title": "Configuring Drill in a Dedicated Cluster", 
-            "next_url": "/docs/configuring-drill-in-a-dedicated-cluster/", 
+            "next_title": "Configuring Drill Memory", 
+            "next_url": "/docs/configuring-drill-memory/", 
             "parent": "Configure Drill", 
             "previous_title": "Configure Drill", 
             "previous_url": "/docs/configure-drill/", 
@@ -1150,7 +1150,7 @@
             "title": "Configuring Connections on Windows", 
             "url": "/docs/configuring-connections-on-windows/"
         }, 
-        "Configuring Drill in a Dedicated Cluster": {
+        "Configuring Drill Memory": {
             "breadcrumbs": [
                 {
                     "title": "Configure Drill", 
@@ -1163,9 +1163,9 @@
             "parent": "Configure Drill", 
             "previous_title": "Configure Drill Introduction", 
             "previous_url": "/docs/configure-drill-introduction/", 
-            "relative_path": "_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md",

-            "title": "Configuring Drill in a Dedicated Cluster", 
-            "url": "/docs/configuring-drill-in-a-dedicated-cluster/"
+            "relative_path": "_docs/configure-drill/020-configuring-drill-memory.md", 
+            "title": "Configuring Drill Memory", 
+            "url": "/docs/configuring-drill-memory/"
         }, 
         "Configuring Multitenant Resources": {
             "breadcrumbs": [
@@ -1301,8 +1301,8 @@
             "next_title": "Configuring a Multitenant Cluster Introduction", 
             "next_url": "/docs/configuring-a-multitenant-cluster-introduction/", 
             "parent": "Configure Drill", 
-            "previous_title": "Configuring Drill in a Dedicated Cluster", 
-            "previous_url": "/docs/configuring-drill-in-a-dedicated-cluster/", 
+            "previous_title": "Configuring Drill Memory", 
+            "previous_url": "/docs/configuring-drill-memory/", 
             "relative_path": "_docs/configure-drill/040-configuring-a-multitenant-cluster.md",

             "title": "Configuring a Multitenant Cluster", 
             "url": "/docs/configuring-a-multitenant-cluster/"
@@ -9273,8 +9273,8 @@
                         }
                     ], 
                     "children": [], 
-                    "next_title": "Configuring Drill in a Dedicated Cluster", 
-                    "next_url": "/docs/configuring-drill-in-a-dedicated-cluster/", 
+                    "next_title": "Configuring Drill Memory", 
+                    "next_url": "/docs/configuring-drill-memory/", 
                     "parent": "Configure Drill", 
                     "previous_title": "Configure Drill", 
                     "previous_url": "/docs/configure-drill/", 
@@ -9295,9 +9295,9 @@
                     "parent": "Configure Drill", 
                     "previous_title": "Configure Drill Introduction", 
                     "previous_url": "/docs/configure-drill-introduction/", 
-                    "relative_path": "_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md",

-                    "title": "Configuring Drill in a Dedicated Cluster", 
-                    "url": "/docs/configuring-drill-in-a-dedicated-cluster/"
+                    "relative_path": "_docs/configure-drill/020-configuring-drill-memory.md",

+                    "title": "Configuring Drill Memory", 
+                    "url": "/docs/configuring-drill-memory/"
                 }, 
                 {
                     "breadcrumbs": [
@@ -9374,8 +9374,8 @@
                     "next_title": "Configuring a Multitenant Cluster Introduction", 
                     "next_url": "/docs/configuring-a-multitenant-cluster-introduction/",

                     "parent": "Configure Drill", 
-                    "previous_title": "Configuring Drill in a Dedicated Cluster", 
-                    "previous_url": "/docs/configuring-drill-in-a-dedicated-cluster/", 
+                    "previous_title": "Configuring Drill Memory", 
+                    "previous_url": "/docs/configuring-drill-memory/", 
                     "relative_path": "_docs/configure-drill/040-configuring-a-multitenant-cluster.md",

                     "title": "Configuring a Multitenant Cluster", 
                     "url": "/docs/configuring-a-multitenant-cluster/"

http://git-wip-us.apache.org/repos/asf/drill/blob/bca47014/_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md b/_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md
deleted file mode 100644
index ec619a9..0000000
--- a/_docs/configure-drill/020-configuring-drill-in-a-dedicated-cluster.md
+++ /dev/null
@@ -1,30 +0,0 @@
----
-title: "Configuring Drill in a Dedicated Cluster"
-parent: "Configure Drill"
----
-
-This section describes how to configure the amount of direct memory allocated to a Drillbit
for query processing in a dedicated Drill cluster. When you use Drill in a cluster with other
workloads, configure memory as described in section, ["Configuring Drill in a Dedicated Cluster"]({{site.baseurl}}/docs/configuring-drill-in-a-dedicated-cluster).

-
-The default memory for a Drillbit is 8G, but Drill prefers 16G or more
-depending on the workload. The total amount of direct memory that a Drillbit
-allocates to query operations cannot exceed the limit set.
-
-Drill mainly uses Java direct memory and performs well when executing
-operations in memory instead of storing the operations on disk. Drill does not
-write to disk unless absolutely necessary, unlike MapReduce where everything
-is written to disk during each phase of a job.
-
-The JVM’s heap memory does not limit the amount of direct memory available in
-a Drillbit. The on-heap memory for Drill is only about 4-8G, which should
-suffice because Drill avoids having data sit in heap memory.
-
-## Modifying Drillbit Memory
-
-You can modify memory for each Drillbit node in your cluster. To modify the
-memory for a Drillbit, edit the `XX:MaxDirectMemorySize` parameter in the
-Drillbit startup script located in `<drill_installation_directory>/conf/drill-
-env.sh`.
-
-{% include startnote.html %}If this parameter is not set, the limit depends on the amount
of available system memory.{% include endnote.html %}
-
-After you edit `<drill_installation_directory>/conf/drill-env.sh`, [restart the Drillbit]({{
site.baseurl }}/docs/starting-drill-in-distributed-mode) on the node.
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/drill/blob/bca47014/_docs/configure-drill/020-configuring-drill-memory.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/020-configuring-drill-memory.md b/_docs/configure-drill/020-configuring-drill-memory.md
new file mode 100644
index 0000000..150ce1f
--- /dev/null
+++ b/_docs/configure-drill/020-configuring-drill-memory.md
@@ -0,0 +1,37 @@
+---
+title: "Configuring Drill Memory"
+parent: "Configure Drill"
+---
+
+This section describes how to configure the amount of direct memory allocated to a Drillbit
for query processing in any Drill cluster, multitenant or not. The default memory for a Drillbit
is 8G, but Drill prefers 16G or more depending on the workload. The total amount of direct
memory that a Drillbit allocates to query operations cannot exceed the limit set.
+
+Drill uses Java direct memory and performs well when executing
+operations in memory instead of storing the operations on disk. Drill does not
+write to disk unless absolutely necessary, unlike MapReduce where everything
+is written to disk during each phase of a job.
+
+The JVM’s heap memory does not limit the amount of direct memory available in
+a Drillbit. The on-heap memory for Drill is typically set at 4-8G (default is 4), which should
+suffice because Drill avoids having data sit in heap memory.
+
+## Modifying Drillbit Memory
+
+You can modify memory for each Drillbit node in your cluster. To modify the
+memory for a Drillbit, edit the `XX:MaxDirectMemorySize` parameter in the
+Drillbit startup script, `drill-env.sh`, located in `<drill_installation_directory>/conf`.
+
+{% include startnote.html %}If XX:MaxDirectMemorySize is not set, the limit depends on the
amount of available system memory.{% include endnote.html %}
+
+After you edit `<drill_installation_directory>/conf/drill-env.sh`, [restart the Drillbit]({{
site.baseurl }}/docs/starting-drill-in-distributed-mode) on the node.
+
+## About the Drillbit startup script
+
+The drill-env.sh file contains the following options:
+
+    DRILL_MAX_DIRECT_MEMORY="8G"
+    DRILL_MAX_HEAP="4G"
+
+    export DRILL_JAVA_OPTS="-Xms1G -Xmx$DRILL_MAX_HEAP -XX:MaxDirectMemorySize=$DRILL_MAX_DIRECT_MEMORY
-XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=1G -ea"
+
+The DRILL_MAX_DIRECT_MEMORY is the Java direct memory. The DRILL_MAX_HEAP is the maximum
theoretical heap limit for the JVM. Xmx specifies the maximum memory allocation pool for a
Java Virtual Machine (JVM). Xms specifies the initial memory allocation pool.
+

http://git-wip-us.apache.org/repos/asf/drill/blob/bca47014/_docs/configure-drill/050-configuring-multitenant-resources.md
----------------------------------------------------------------------
diff --git a/_docs/configure-drill/050-configuring-multitenant-resources.md b/_docs/configure-drill/050-configuring-multitenant-resources.md
index 9a944e8..53341fb 100644
--- a/_docs/configure-drill/050-configuring-multitenant-resources.md
+++ b/_docs/configure-drill/050-configuring-multitenant-resources.md
@@ -2,7 +2,29 @@
 title: "Configuring Multitenant Resources"
 parent: "Configuring a Multitenant Cluster"
 ---
-Drill operations are memory and CPU-intensive. You need to statically partition the cluster
to designate which partition handles which workload. To configure resources for Drill in a
MapR cluster, modify one or more of the following files in `/opt/mapr/conf/conf.d` that the
installation process creates. 
+Drill operations are memory and CPU-intensive. Currently, Drill resources are managed outside
MapR Warden service in terms of configuring the resources as well as enforcing the resource
usage within the limit. Configure memory for Drill in a multitenant by modifying drill-env.sh.
<Add detail on property names etc>
+
+3. To ensure Warden account for resources required for Drill, make sure the following properties
are set appropriately in warden.drill-bits.conf. For reference on the meaning of the properties
refer to the following MapR doc <add link>
+
+service.heapsize.min
+service.heapsize.max
+service.heapsize.percent
+
+The percent should always be considered as the one to change, it is more intuitive to understand
and grows/shrinks according to different node's configuration. 
+
+It will be good if someone in Drill QA could try it out and see if it fits Drill's needs.

+
+ 
+
+Note that these properties should be set in addition to configuring the resources the in
drill-env.conf even if you didnt change the defaults in drill-env.sh. Setting up memory limit
in drill-env.sh tells Drill how much resources to use during query execution and setting up
these warden-drill-bits.conf tells warden not to commit these resources to some other process.
In near future, we expect to provide a more deeper integration on these settings
+\<give an example>
+
+4. This configuration is same whether you use Drill is enabled cluster or non-YARN cluster.
+
+
+
+
+You need to statically partition the cluster to designate which partition handles which workload.
To configure resources for Drill in a MapR cluster, modify one or more of the following files
in `/opt/mapr/conf/conf.d` that the installation process creates. 
 
 * `warden.drill-bits.conf`
 * `warden.nodemanager.conf`


Mime
View raw message