hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wan...@apache.org
Subject hadoop git commit: YARN-7056. Document Resource Profiles feature. (Sunil G via wangda)
Date Thu, 07 Sep 2017 17:08:07 GMT
Repository: hadoop
Updated Branches:
  refs/heads/YARN-3926 eaab1364c -> 9698083fd


YARN-7056. Document Resource Profiles feature. (Sunil G via wangda)

Change-Id: I5f5d52cdf7aba4ac09684aab573e6916dbad183e


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/9698083f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/9698083f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/9698083f

Branch: refs/heads/YARN-3926
Commit: 9698083fd6313d6a206e62cfc8e6cce6b156d84a
Parents: eaab136
Author: Wangda Tan <wangda@apache.org>
Authored: Thu Sep 7 10:07:35 2017 -0700
Committer: Wangda Tan <wangda@apache.org>
Committed: Thu Sep 7 10:07:35 2017 -0700

----------------------------------------------------------------------
 hadoop-project/src/site/site.xml                |   1 +
 .../src/site/markdown/ResourceProfiles.md       | 116 +++++++++++++++++++
 2 files changed, 117 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/9698083f/hadoop-project/src/site/site.xml
----------------------------------------------------------------------
diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml
index e7af227..4685e2a 100644
--- a/hadoop-project/src/site/site.xml
+++ b/hadoop-project/src/site/site.xml
@@ -139,6 +139,7 @@
       <item name="Graceful Decommission" href="hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html"/>
       <item name="Opportunistic Containers" href="hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html"/>
       <item name="YARN Federation" href="hadoop-yarn/hadoop-yarn-site/Federation.html"/>
+      <item name="Resource Profiles" href="hadoop-yarn/hadoop-yarn-site/ResourceProfiles.html"/>
     </menu>
 
     <menu name="YARN REST APIs" inherit="top">

http://git-wip-us.apache.org/repos/asf/hadoop/blob/9698083f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md
new file mode 100644
index 0000000..d422ab6
--- /dev/null
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md
@@ -0,0 +1,116 @@
+<!--
+   Licensed to the Apache Software Foundation (ASF) under one or more
+   contributor license agreements.  See the NOTICE file distributed with
+   this work for additional information regarding copyright ownership.
+   The ASF licenses this file to You under the Apache License, Version 2.0
+   (the "License"); you may not use this file except in compliance with
+   the License.  You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+-->
+
+Hadoop: YARN Resource Profiles
+==============================
+
+Overview
+--------
+Resource profiles support in YARN helps to extend the YARN resource model to a more flexible
model which makes it easier to add new countable resource­types. This solution also helps
the users to submit jobs with ease to specify the resources they need.
+
+Resource model of YARN
+-----------------------
+Resource Manager will load a new configuration file named `resource-types.xml` to determine
the set of resource ­types for which scheduling is enabled. Sample XML will look like below.
+
+```xml
+<configuration>
+  <property>
+    <name>yarn.resource-types</name>
+    <value>resource1, resource2</value>
+  </property>
+
+  <property>
+    <name>yarn.resource-types.resource1.units</name>
+    <value>G</value>
+  </property>
+</configuration>
+```
+
+Similarly, a new configuration file `node­-resources.xml` will also be loaded by Node Manager
where the resource capabilities of a node can be specified.
+
+```xml
+<configuration>
+ <property>
+   <name>yarn.nodemanager.resource-type.resource1</name>
+   <value>5G</value>
+ </property>
+
+ <property>
+   <name>yarn.nodemanager.resource-type.resource2</name>
+   <value>2m</value>
+ </property>
+
+</configuration>
+```
+
+Node Manager will use these custom resource types and will register it's capability to Resource
Manager.
+
+Resource Profiles
+-----------------
+As per the updated resource model of YARN, specifying and configuring additional resource
types for container allocation request will become more cumbersome. With the support of resource
profiles, such requests could be placed by using simple profile names which has defined meaning
from YARN.
+
+Admin could specify `resource-profiles.json` which indicates various profiles supported.
+
+```json
+{
+    "minimum": {
+        "memory-mb" : 1024,
+        "vcores" : 1
+    },
+    "default" : {
+        "memory-mb" : 2048,
+        "vcores" : 2
+    },
+    "maximum" : {
+        "memory-mb": 4096,
+        "vcores" : 4
+    },
+    "http" : {
+        "memory-mb" : 2048,
+        "vcores" : 2
+    }
+}
+```
+This indicates that profile named "minimum", "default", "maximum" and "http" is supported.
Clients could easily use this name instead of specifying each resource details.
+
+Configurations
+-------------
+
+*In `yarn-site.xml`*
+
+| Configuration Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.resource-profiles.enabled` | *(Required)* In the server side it indicates
whether resource profiles support is enabled or not. Defaults to `false`. |
+
+Please note that, `resource-types.xml` and `node­-resources.xml` file also need to be placed
in conf directory if new resources are to be added to YARN.
+
+*In `resource-types.xml`*
+
+| Configuration Property | Value | Description |
+|:---- |:---- |:---- |
+| `yarn.resource-types` | resource1 | Custom resource  |
+| `yarn.resource-types.resource1.units` | G | Default unit for resource1 type  |
+
+*In `node­-resources.xml`*
+
+| Configuration Property | Value | Description |
+|:---- |:---- |:---- |
+| `yarn.nodemanager.resource-type.resource1` | 5G | Resource capability for resource named
'resource1'. |
+
+Use it
+--------
+Distributed Shell is supporting this feature and user could specify a resource profile name
which YARN is supporting while running a DS application.


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message