drill-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tshi...@apache.org
Subject [06/17] drill git commit: fix links
Date Sat, 02 May 2015 06:28:32 GMT
fix links


Project: http://git-wip-us.apache.org/repos/asf/drill/repo
Commit: http://git-wip-us.apache.org/repos/asf/drill/commit/d56fcb40
Tree: http://git-wip-us.apache.org/repos/asf/drill/tree/d56fcb40
Diff: http://git-wip-us.apache.org/repos/asf/drill/diff/d56fcb40

Branch: refs/heads/gh-pages
Commit: d56fcb40fcba45afa9fbaa716c4ba5014aa4d9dd
Parents: bcc1bc0
Author: Kristine Hahn <khahn@maprtech.com>
Authored: Fri May 1 11:24:01 2015 -0700
Committer: Kristine Hahn <khahn@maprtech.com>
Committed: Fri May 1 11:24:01 2015 -0700

----------------------------------------------------------------------
 _data/docs.json                                 |  8 +-
 ...nfiguring-dfferent-workloads-introduction.md |  2 +-
 ...-configuring-resources-in-a-mixed-cluster.md | 80 ++++++++++++++++++++
 3 files changed, 85 insertions(+), 5 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/drill/blob/d56fcb40/_data/docs.json
----------------------------------------------------------------------
diff --git a/_data/docs.json b/_data/docs.json
index 6805349..83fb2b4 100644
--- a/_data/docs.json
+++ b/_data/docs.json
@@ -881,7 +881,7 @@
                     "parent": "Configuring Different Workloads", 
                     "previous_title": "Configuring Different Workloads Introduction", 
                     "previous_url": "/docs/configuring-different-workloads-introduction/",

-                    "relative_path": "_docs/manage-drill/015-configuring-drill-in-a-cluster.md",

+                    "relative_path": "_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md",

                     "title": "Configuring Resources in a Mixed Cluster", 
                     "url": "/docs/configuring-resources-in-a-mixed-cluster/"
                 }, 
@@ -992,7 +992,7 @@
             "parent": "Configuring Different Workloads", 
             "previous_title": "Configuring Different Workloads Introduction", 
             "previous_url": "/docs/configuring-different-workloads-introduction/", 
-            "relative_path": "_docs/manage-drill/015-configuring-drill-in-a-cluster.md",

+            "relative_path": "_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md",

             "title": "Configuring Resources in a Mixed Cluster", 
             "url": "/docs/configuring-resources-in-a-mixed-cluster/"
         }, 
@@ -3530,7 +3530,7 @@
                             "parent": "Configuring Different Workloads", 
                             "previous_title": "Configuring Different Workloads Introduction",

                             "previous_url": "/docs/configuring-different-workloads-introduction/",

-                            "relative_path": "_docs/manage-drill/015-configuring-drill-in-a-cluster.md",

+                            "relative_path": "_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md",

                             "title": "Configuring Resources in a Mixed Cluster", 
                             "url": "/docs/configuring-resources-in-a-mixed-cluster/"
                         }, 
@@ -10958,7 +10958,7 @@
                             "parent": "Configuring Different Workloads", 
                             "previous_title": "Configuring Different Workloads Introduction",

                             "previous_url": "/docs/configuring-different-workloads-introduction/",

-                            "relative_path": "_docs/manage-drill/015-configuring-drill-in-a-cluster.md",

+                            "relative_path": "_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md",

                             "title": "Configuring Resources in a Mixed Cluster", 
                             "url": "/docs/configuring-resources-in-a-mixed-cluster/"
                         }, 

http://git-wip-us.apache.org/repos/asf/drill/blob/d56fcb40/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md b/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md
index efbbb2e..4bd1664 100644
--- a/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md
+++ b/_docs/manage-drill/013-configuring-dfferent-workloads-introduction.md
@@ -16,7 +16,7 @@ Drill typically runs along side other workloads, including the following:
 You need to plan and configure these resources for use with Drill and other workloads: 
 
 * [Memory]({{site.baseurl}}/docs/configuring-resources-in-a-mixed-cluster)  
-* [CPU]({{site.baseurl}}/docs/configuring-drill-in-a-mixed-cluster#how-to-manage-drill-cpu-resources)
 
+* [CPU]({{site.baseurl}}/docs/configuring-resources-in-a-mixed-cluster#how-to-manage-drill-cpu-resources)
 
 * Disk  
 
 Configure, memory, queues, and parallelization when users [share a Drillbit]({{site.baseurl}}/docs/configuring-resources-for-a-shared-drillbit).
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/drill/blob/d56fcb40/_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md
----------------------------------------------------------------------
diff --git a/_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md b/_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md
new file mode 100644
index 0000000..474d123
--- /dev/null
+++ b/_docs/manage-drill/015-configuring-resources-in-a-mixed-cluster.md
@@ -0,0 +1,80 @@
+---
+title: "Configuring Resources in a Mixed Cluster"
+parent: "Configuring Different Workloads"
+---
+Drill operations are memory and CPU-intensive. You need to statically partition the cluster
to designate which partition handles which workload. To configure resources for Drill in a
MapR cluster, modify one or more of the following files in `/opt/mapr/conf/conf.d` that the
installation process creates. 
+
+* `warden.drill-bits.conf`
+* `warden.nodemanager.conf`
+* `warden.resourcemanager.conf`
+
+Configure Drill memory by modifying `warden.drill-bits.conf` in YARN and non-YARN clusters.
Configure other resources by modifying `warden.nodemanager.conf `and `warden.resourcemanager.conf
`in a YARN-enabled cluster.
+
+## Configuring Drill Memory in a Mixed Cluster
+
+Add the following lines to the `warden.drill-bits.conf` file to configure memory resources
for Drill:
+
+    service.heapsize.min=<some value in MB>
+    service.heapsize.max=<some value in MB>
+    service.heapsize.percent=<a whole number>
+
+The service.heapsize.percent is the percentage of memory for the service bounded by minimum
and maximum values.
+
+## Configuring Drill in a YARN-enabled MapR Cluster
+
+To add Drill to a YARN-enabled cluster, change memory resources to suit your application.
For example, you have 120G of available memory that you allocate to following workloads in
a Yarn-enabled cluster:
+
+File system = 20G  
+HBase = 20G  
+Yarn = 20G  
+OS = 8G  
+
+If Yarn does most of the work, give Drill 20G, for example, and give Yarn 60G. If you expect
a heavy query load, give Drill 60G and Drill 20G.
+
+{% include startnote.html %}Drill will execute queries within Yarn soon.{% include endnote.html
%} [DRILL-142](https://issues.apache.org/jira/browse/DRILL-142)
+
+YARN consists of two main services:
+
+* ResourceManager  
+  There is at least one instance in a cluster, more if you configure high availability. 

+* NodeManager  
+  There is one instance per node. 
+
+ResourceManager and NodeManager memory in `warden.resourcemanager.conf` and
+ `warden.nodemanager.conf` are set to the following defaults. 
+
+    service.heapsize.min=64
+    service.heapsize.max=325
+    service.heapsize.percent=2
+
+Change these settings for NodeManager and ResourceManager to reconfigure the total memory
required for YARN services to run. If you want to place an upper limit on memory set YARN_NODEMANAGER_HEAPSIZE
or YARN_RESOURCEMANAGER_HEAPSIZE environment variable in /opt/mapr/hadoop/hadoop-2.5.1/etc/hadoop/yarn-env.sh.
The -Xmx option is not set, allowing memory on to grow as needed.
+
+### MapReduce v1 Resources
+
+The following default settings in /opt/mapr/conf/warden.conf control MapReduce v1 memory:
+
+    mr1.memory.percent=50
+    mr1.cpu.percent=50
+    mr1.disk.percent=50
+
+Modify these settings to reconfigure MapReduce v1 resources to suit your application needs,
as described in section ["Resource Allocation for Jobs and Applications"](http://doc.mapr.com/display/MapR/Resource+Allocation+for+Jobs+and+Applications)
of the MapR documentation. Remaining memory is given to YARN applications. 
+
+
+### MapReduce v2 and other Resources
+
+You configure memory for each service by setting three values in `warden.conf`.
+
+    service.command.<servicename>.heapsize.percent
+    service.command.<servicename>.heapsize.max
+    service.command.<servicename>.heapsize.min
+
+Configure memory for other services in the same manner, as described in [MapR documentation](http://doc.mapr.com/display/MapR/warden.%3Cservicename%3E.conf)
+
+For more information about managing memory in a MapR cluster, see the following sections
in the MapR documentation:
+
+* [Memory Allocation for Nodes](http://doc.mapr.com/display/MapR40x/Memory+Allocation+for+Nodes)
 
+* [Cluster Resource Allocation](http://doc.mapr.com/display/MapR40x/Cluster+Resource+Allocation)
 
+* [Customizing Memory Settings for MapReduce v1](http://doc.mapr.com/display/MapR40x/Customize+Memory+Settings+for+MapReduce+v1)
 
+
+## How to Manage Drill CPU Resources
+Currently, you do not manage CPU resources within Drill. [Use Linux `cgroups`](http://en.wikipedia.org/wiki/Cgroups)
to manage the CPU resources.
\ No newline at end of file


Mime
View raw message