hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jh...@apache.org
Subject [15/17] hadoop git commit: YARN-7238. Documentation for API based scheduler configuration management
Date Thu, 05 Oct 2017 20:48:04 GMT
YARN-7238. Documentation for API based scheduler configuration management


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/8e75b9d8
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/8e75b9d8
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/8e75b9d8

Branch: refs/heads/YARN-5734-branch-2
Commit: 8e75b9d8a6d955f8bd28186e6ca21ab938f09f05
Parents: 25e2aae
Author: Jonathan Hung <jhung@linkedin.com>
Authored: Mon Sep 25 17:47:32 2017 -0700
Committer: Jonathan Hung <jhung@linkedin.com>
Committed: Thu Oct 5 13:44:06 2017 -0700

----------------------------------------------------------------------
 .../src/site/markdown/CapacityScheduler.md      |  24 ++-
 .../src/site/markdown/ResourceManagerRest.md    | 186 +++++++++++++++++++
 .../src/site/markdown/YarnCommands.md           |  13 ++
 3 files changed, 221 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/8e75b9d8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 6bb8489..d70f891 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -295,9 +295,29 @@ The `ReservationSystem` is integrated with the `CapacityScheduler` queue
hierach
 Changing Queue Configuration
 ----------------------------
 
-Changing queue properties and adding new queues is very simple. You need to edit **conf/capacity-scheduler.xml**
and run *yarn rmadmin -refreshQueues*.
+Changing queue/scheduler properties and adding/removing queues can be done in two ways, via
file or via API.
+
+### Changing queue configuration via file
+
+  To edit by file, you need to edit **conf/capacity-scheduler.xml** and run *yarn rmadmin
-refreshQueues*.
 
     $ vi $HADOOP_CONF_DIR/capacity-scheduler.xml
     $ $HADOOP_YARN_HOME/bin/yarn rmadmin -refreshQueues
 
-**Note:** Queues cannot be *deleted*, only addition of new queues is supported - the updated
queue configuration should be a valid one i.e. queue-capacity at each *level* should be equal
to 100%.
+### Changing queue configuration via API
+
+  Editing by API uses a backing store for the scheduler configuration. To enable this, the
following parameters can be configured in yarn-site.xml.
+
+  | Property | Description |
+  |:---- |:---- |
+  | `yarn.scheduler.capacity.configuration.provider` | The type of configuration provider
to use for capacity scheduler. To enable changing queue configuration via API, this should
be set to *store*. Default value is *file*, which disables the API and reverts back to changing
queue configuration via file. |
+  | `yarn.scheduler.configuration.store.class` | The type of backing store to use. Default
value is *memory*, which stores the scheduler configuration in memory (and does not persist
configuration changes across restarts). Other values are *leveldb* (using a leveldb-based
implementation), and *zk* (using a zookeeper-based implementation). |
+  | `yarn.scheduler.configuration.mutation.acl-policy.class` | An ACL policy can be configured
to restrict which users can modify which queues. Default value is *org.apache.hadoop.yarn.server.resourcemanager.scheduler.DefaultConfigurationMutationACLPolicy*,
which only allows YARN admins to make any configuration modifications. Another value is *org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.QueueAdminConfigurationMutationACLPolicy*,
which only allows queue modifications if the caller is an admin of the queue. |
+  | `yarn.scheduler.configuration.store.max-logs` | Configuration changes are audit logged
in the backing store, if using leveldb or zookeeper. This configuration controls the maximum
number of audit logs to store, dropping the oldest logs when exceeded. Default is 1000. |
+  | `yarn.scheduler.configuration.leveldb-store.path` | The storage path of the configuration
store when using leveldb. Default value is *${hadoop.tmp.dir}/yarn/system/confstore*. |
+  | `yarn.scheduler.configuration.leveldb-store.compaction-interval-secs` | The interval
for compacting the configuration store in seconds, when using leveldb. Default value is 86400,
or one day. |
+  | `yarn.scheduler.configuration.zk-store.parent-path` | The zookeeper root node path for
configuration store related information, when using zookeeper. Default value is */confstore*.
|
+
+  **Note:** When enabling backing store for scheduler configuration, *yarn rmadmin -refreshQueues*
will be disabled, i.e. it will no longer be possible to update configuration via file.
+
+  See the [YARN Resource Manager REST API](ResourceManagerRest.html#Scheduler_Configuration_Mutation_API)
for examples on how to change scheduler configuration via REST, and [YARN Commands Reference](YarnCommands.html#schedconf)
for examples on how to change scheduler configuration via command line.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8e75b9d8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
index 7c1f418..2ead7a0 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
@@ -4430,3 +4430,189 @@ Response Body:
     <remainingTimeInSeconds>90</remainingTimeInSeconds>
 </timeout>
 ```
+
+Scheduler Configuration Mutation API
+--------------------------------
+
+The scheduler configuration mutation API provides a way to modify scheduler/queue configuration
and queue hierarchy.
+
+
+### URI
+
+      * http://rm-http-address:port/ws/v1/cluster/sched-conf
+
+### HTTP Operations Supported
+
+      * PUT
+
+### Elements of the *schedConf* object
+
+| Item | Data Type | Description |
+|:---- |:---- |:---- |
+| update-queue | object | A queue whose configurations should be updated |
+| add-queue | object | A queue to add to the scheduler along with this queue's configurations
|
+| remove-queue | string | Full path name of a queue to remove |
+| global-updates | map | Map of key value pairs to update scheduler's global configuration
|
+
+### PUT Request Examples
+
+Put requests are used to modify the scheduler configuration. A successful mutation results
in a 200 response. A malformed request or one which resulted in an invalid scheduler configuration
results in a 400 response.
+
+**Updating queue configuration(s)**
+
+Request for updating queue configurations.
+
+*Elements of the* update-queue *object*
+
+| Item | Data Type | Description |
+|:---- |:---- |:---- |
+| queueName | string | Full path name of the queue to update |
+| params | map | A map of key value configuration pairs to update for this queue |
+
+Assuming we are using the capacity scheduler and the current queue configuration is a single
queue *root.default*, this example sets *root.default*'s maximum applications to 100 and its
minimum user limit percent to 10.
+
+HTTP Request:
+
+```xml
+      Accept: application/xml
+      PUT http://rm-http-address:port/ws/v1/cluster/sched-conf
+      Content-Type: application/xml
+      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+      <schedConf>
+        <update-queue>
+          <queueName>root.default</queueName>
+          <params>
+            <entry>
+              <key>maximum-applications</key>
+              <value>100</value>
+            </entry>
+            <entry>
+              <key>minimum-user-limit-percent</key>
+              <value>10</value>
+            </entry>
+          </params>
+        </update-queue>
+      </schedConf>
+```
+
+
+Response Header:
+
+      HTTP/1.1 200 OK
+      Content-Type: application/xml
+      Transfer-Encoding: chunked
+
+
+**Adding a queue**
+
+Request for adding queues/updating queue configurations.
+
+*Elements of the* add-queue *object*
+
+| Item | Data Type | Description |
+|:---- |:---- |:---- |
+| queueName | string | Full path name of the queue to add |
+| params | map | A map of key value configuration pairs to set for this queue |
+
+Assuming we are using the capacity scheduler and the current queue configuration is a single
queue *root.default*, this example adds a queue *root.a* with capacity/maximum-capacity 10,
and adjusts *root.default*'s capacity/maximum-capacity to 90. (More complex examples include
adding a queue whose parent is also being added in the same request, or adding multiple sibling
queues.)
+
+HTTP Request:
+
+```xml
+      Accept: application/xml
+      PUT http://rm-http-address:port/ws/v1/cluster/sched-conf
+      Content-Type: application/xml
+      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+      <schedConf>
+        <add-queue>
+          <queueName>root.a</queueName>
+          <params>
+            <entry>
+              <key>capacity</key>
+              <value>10</value>
+            </entry>
+            <entry>
+              <key>maximum-capacity</key>
+              <value>10</value>
+            </entry>
+          </params>
+        </add-queue>
+        <update-queue>
+          <queueName>root.default</queueName>
+          <params>
+            <entry>
+              <key>capacity</key>
+              <value>90</value>
+            </entry>
+            <entry>
+              <key>maximum-capacity</key>
+              <value>90</value>
+            </entry>
+          </params>
+        </update-queue>
+      </schedConf>
+```
+
+
+Response Header:
+
+      HTTP/1.1 200 OK
+      Content-Type: application/xml
+      Transfer-Encoding: chunked
+
+**Removing queues**
+
+Request for removing queues from the queue hierarchy.
+
+Assuming we are using the capacity scheduler and the current queue configuration is three
queues *root.default*, *root.a*, and *root.b*, this example removes both *root.a* and *root.b*.
(More complex examples include removing a parent queue and its children.)
+
+**Note:** Queues must be put into `STOPPED` state before they are deleted. Any updated queue
configuration should be a valid one i.e. queue-capacity at each *level* should be equal to
100%.
+
+
+HTTP Request:
+
+```xml
+      Accept: application/xml
+      PUT http://rm-http-address:port/ws/v1/cluster/sched-conf
+      Content-Type: application/xml
+      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+      <schedConf>
+        <remove-queue>root.a</remove-queue>
+        <remove-queue>root.b</remove-queue>
+      </schedConf>
+```
+
+
+Response Header:
+
+      HTTP/1.1 200 OK
+      Content-Type: application/xml
+      Transfer-Encoding: chunked
+
+**Updating global scheduler configurations**
+
+Request for updating global scheduler configurations. Assuming we are using the capacity
scheduler, this example enables queue mappings. For global configuration updates, the full
configuration key must be specified.
+
+HTTP Request:
+
+```xml
+      Accept: application/xml
+      PUT http://rm-http-address:port/ws/v1/cluster/sched-conf
+      Content-Type: application/xml
+      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+      <schedConf>
+        <global-updates>
+          <entry>
+            <key>yarn.scheduler.capacity.queue-mappings-override.enable</key>
+            <value>true</value>
+          </entry>
+        </global-updates>
+      </schedConf>
+```
+
+
+Response Header:
+
+      HTTP/1.1 200 OK
+      Content-Type: application/xml
+      Transfer-Encoding: chunked

http://git-wip-us.apache.org/repos/asf/hadoop/blob/8e75b9d8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
index 1b36619..b110197 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md
@@ -238,6 +238,19 @@ Usage:
 
 Runs ResourceManager admin client
 
+### schedconf
+
+Usage: `yarn schedconf [options]`
+
+| COMMAND\_OPTIONS | Description |
+|:---- |:---- |
+| -add <queuePath1,key1=val1,key2=val2;queuePath2,key3=val3> | Semicolon separated
values of queues to add and their queue configurations. This example adds queue "queuePath1"
(a full path name), which has queue configurations key1=val1 and key2=val2. It also adds queue
"queuePath2", which has queue configuration key3=val3. |
+| -remove <queuePath1,queuePath2> | Comma-separated queues to remove. This example
removes queuePath1 and queuePath2 queues (full path names). **Note:** Queues must be put into
`STOPPED` state before they are deleted. |
+| -update <queuePath1,key1=val1,key2=val2;queuePath2,key3=val3> | Semicolon separated
values of queues whose configurations should be updated. This example sets key1=val1 and key2=val2
for queue configuration of queuePath1 (full path name), and sets key3=val3 for queue configuration
of queuePath2. |
+| -global <key1=val1,key2=val2> | Update scheduler global configurations. This example
sets key1=val1 and key2=val2 for scheduler's global configuration. |
+
+Updates scheduler configuration
+
 ### scmadmin
 
 Usage: `yarn scmadmin [options] `


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message