hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wan...@apache.org
Subject [19/50] [abbrv] hadoop git commit: YARN-7533. Documentation for absolute resource support in Capacity Scheduler. Contributed by Sunil G.
Date Sun, 10 Dec 2017 21:54:52 GMT
YARN-7533. Documentation for absolute resource support in Capacity Scheduler. Contributed by
Sunil G.


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/532bbf46
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/532bbf46
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/532bbf46

Branch: refs/heads/YARN-6592
Commit: 532bbf4602e418276961cfc4f6b5e3f70e9cedc0
Parents: 4db4a4a
Author: Sunil G <sunilg@apache.org>
Authored: Wed Nov 29 22:57:54 2017 +0530
Committer: Wangda Tan <wangda@apache.org>
Committed: Thu Dec 7 18:56:54 2017 -0800

----------------------------------------------------------------------
 .../src/site/markdown/CapacityScheduler.md                | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/532bbf46/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 0449e88..86d9167 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -64,6 +64,8 @@ The `CapacityScheduler` supports the following features:
 
 * **Priority Scheduling** - This feature allows applications to be submitted and scheduled
with different priorities. Higher integer value indicates higher priority for an application.
Currently Application priority is supported only for FIFO ordering policy.
 
+* **Absolute Resource Configuration** - Administrators could specify absolute resources to
a queue instead of providing percentage based values. This provides better control for admins
to configure required amount of resources for a given queue.
+
 Configuration
 -------------
 
@@ -118,14 +120,18 @@ Configuration
 
 | Property | Description |
 |:---- |:---- |
-| `yarn.scheduler.capacity.<queue-path>.capacity` | Queue *capacity* in percentage
(%) as a float (e.g. 12.5). The sum of capacities for all queues, at each level, must be equal
to 100. Applications in the queue may consume more resources than the queue's capacity if
there are free resources, providing elasticity. |
-| `yarn.scheduler.capacity.<queue-path>.maximum-capacity` | Maximum queue capacity
in percentage (%) as a float. This limits the *elasticity* for applications in the queue.
Defaults to -1 which disables it. |
+| `yarn.scheduler.capacity.<queue-path>.capacity` | Queue *capacity* in percentage
(%) as a float (e.g. 12.5) OR as absolute resource queue minimum capacity. The sum of capacities
for all queues, at each level, must be equal to 100. However if absolute resource is configured,
sum of absolute resources of child queues could be less than it's parent absolute resource
capacity. Applications in the queue may consume more resources than the queue's capacity if
there are free resources, providing elasticity. |
+| `yarn.scheduler.capacity.<queue-path>.maximum-capacity` | Maximum queue capacity
in percentage (%) as a float OR as absolute resource queue maximum capacity. This limits the
*elasticity* for applications in the queue. Defaults to -1 which disables it. |
 | `yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent` | Each queue enforces
a limit on the percentage of resources allocated to a user at any given time, if there is
demand for resources. The user limit can vary between a minimum and maximum value. The former
(the minimum value) is set to this property value and the latter (the maximum value) depends
on the number of users who have submitted applications. For e.g., suppose the value of this
property is 25. If two users have submitted applications to a queue, no single user can use
more than 50% of the queue resources. If a third user submits an application, no single user
can use more than 33% of the queue resources. With 4 or more users, no user can use more than
25% of the queues resources. A value of 100 implies no user limits are imposed. The default
is 100. Value is specified as a integer. |
 | `yarn.scheduler.capacity.<queue-path>.user-limit-factor` | The multiple of the queue
capacity which can be configured to allow a single user to acquire more resources. By default
this is set to 1 which ensures that a single user can never take more than the queue's configured
capacity irrespective of how idle the cluster is. Value is specified as a float. |
 | `yarn.scheduler.capacity.<queue-path>.maximum-allocation-mb` | The per queue maximum
limit of memory to allocate to each container request at the Resource Manager. This setting
overrides the cluster configuration `yarn.scheduler.maximum-allocation-mb`. This value must
be smaller than or equal to the cluster maximum. |
 | `yarn.scheduler.capacity.<queue-path>.maximum-allocation-vcores` | The per queue
maximum limit of virtual cores to allocate to each container request at the Resource Manager.
This setting overrides the cluster configuration `yarn.scheduler.maximum-allocation-vcores`.
This value must be smaller than or equal to the cluster maximum. |
 | `yarn.scheduler.capacity.<queue-path>.user-settings.<user-name>.weight` | This
floating point value is used when calculating the user limit resource values for users in
a queue. This value will weight each user more or less than the other users in the queue.
For example, if user A should receive 50% more resources in a queue than users B and C, this
property will be set to 1.5 for user A.  Users B and C will default to 1.0. |
 
+  * Resource Allocation using Absolute Resources configuration
+
+ `CapacityScheduler` supports configuration of absolute resources instead of providing Queue
*capacity* in percentage. As mentioned in above configuration section for `yarn.scheduler.capacity.<queue-path>.capacity`
and `yarn.scheduler.capacity.<queue-path>.max-capacity`, administrator could specify
an absolute resource value like `[memory=10240,vcores=12]`. This is a valid configuration
which indicates 10GB Memory and 12 VCores.
+
   * Running and Pending Application Limits
   
   The `CapacityScheduler` supports the following parameters to control the running and pending
applications:


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message