hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jl...@apache.org
Subject hadoop git commit: YARN-4492. Add documentation for preemption supported in Capacity scheduler. Contributed by Naganarasimha G R
Date Thu, 21 Jan 2016 15:44:11 GMT
Repository: hadoop
Updated Branches:
  refs/heads/branch-2.7 e160ddfb8 -> bade7f06e


YARN-4492. Add documentation for preemption supported in Capacity scheduler. Contributed by
Naganarasimha G R


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/bade7f06
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/bade7f06
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/bade7f06

Branch: refs/heads/branch-2.7
Commit: bade7f06e80ba7ca035c9b1b5d910acdc3cdbf19
Parents: e160ddf
Author: Jason Lowe <jlowe@apache.org>
Authored: Thu Jan 21 15:43:23 2016 +0000
Committer: Jason Lowe <jlowe@apache.org>
Committed: Thu Jan 21 15:43:23 2016 +0000

----------------------------------------------------------------------
 hadoop-yarn-project/CHANGES.txt                 |  3 ++
 .../src/site/markdown/CapacityScheduler.md      | 31 +++++++++++++++++++-
 2 files changed, 33 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/bade7f06/hadoop-yarn-project/CHANGES.txt
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt
index b4fe340..e18b8e7 100644
--- a/hadoop-yarn-project/CHANGES.txt
+++ b/hadoop-yarn-project/CHANGES.txt
@@ -10,6 +10,9 @@ Release 2.7.3 - UNRELEASED
 
     YARN-4287. Capacity Scheduler: Rack Locality improvement. (Nathan Roberts via wangda)
 
+    YARN-4492. Add documentation for preemption supported in Capacity
+    scheduler (Naganarasimha G R via jlowe)
+
   OPTIMIZATIONS
 
   BUG FIXES

http://git-wip-us.apache.org/repos/asf/hadoop/blob/bade7f06/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 9227b1f..7b19acd 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -22,6 +22,7 @@ Hadoop: Capacity Scheduler
     * [Setting up `ResourceManager` to use `CapacityScheduler`](#Setting_up_ResourceManager_to_use_CapacityScheduler`)
     * [Setting up queues](#Setting_up_queues)
     * [Queue Properties](#Queue_Properties)
+    * [Capacity Scheduler container preemption](#Capacity_Scheduler_container_preemption)
     * [Other Properties](#Other_Properties)
     * [Reviewing the configuration of the CapacityScheduler](#Reviewing_the_configuration_of_the_CapacityScheduler)
 * [Changing Queue Configuration](#Changing_Queue_Configuration)
@@ -57,7 +58,7 @@ The `CapacityScheduler` supports the following features:
 
 * **Security** - Each queue has strict ACLs which controls which users can submit applications
to individual queues. Also, there are safe-guards to ensure that users cannot view and/or
modify applications from other users. Also, per-queue and system administrator roles are supported.
 
-* **Elasticity** - Free resources can be allocated to any queue beyond it's capacity. When
there is demand for these resources from queues running below capacity at a future point in
time, as tasks scheduled on these resources complete, they will be assigned to applications
on queues running below the capacity (pre-emption is not supported). This ensures that resources
are available in a predictable and elastic manner to queues, thus preventing artifical silos
of resources in the cluster which helps utilization.
+* **Elasticity** - Free resources can be allocated to any queue beyond its capacity. When
there is demand for these resources from queues running below capacity at a future point in
time, as tasks scheduled on these resources complete, they will be assigned to applications
on queues running below the capacity (pre-emption is also supported). This ensures that resources
are available in a predictable and elastic manner to queues, thus preventing artifical silos
of resources in the cluster which helps utilization.
 
 * **Multi-tenancy** - Comprehensive set of limits are provided to prevent a single application,
user and queue from monopolizing resources of the queue or the cluster as a whole to ensure
that the cluster isn't overwhelmed.
 
@@ -177,6 +178,34 @@ Example:
  </property>
 ```
 
+
+### Capacity Scheduler container preemption
+
+ The `CapacityScheduler` supports preemption of container from the queues whose resource
usage is more than their guaranteed capacity. The following configuration parameters need
to be enabled in yarn-site.xml for supporting preemption of application containers.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.scheduler.monitor.enable` | Enable a set of periodic monitors (specified
in yarn.resourcemanager.scheduler.monitor.policies) that affect the scheduler. Default value
is false. |
+| `yarn.resourcemanager.scheduler.monitor.policies` | The list of SchedulingEditPolicy classes
that interact with the scheduler. Configured policies need to be compatible with the scheduler.
Default value is `org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy`
which is compatible with `CapacityScheduler` |
+
+The following configuration parameters can be configured in yarn-site.xml to control the
preemption of containers when `ProportionalCapacityPreemptionPolicy` class is configured for
`yarn.resourcemanager.scheduler.monitor.policies`
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.monitor.capacity.preemption.observe_only` | If true, run the policy
but do not affect the cluster with preemption and kill events. Default value is false |
+| `yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval` | Time in milliseconds
between invocations of this ProportionalCapacityPreemptionPolicy policy. Default value is
3000 |
+| `yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill` | Time in milliseconds
between requesting a preemption from an application and killing the container. Default value
is 15000 |
+| `yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round` | Maximum
percentage of resources preempted in a single round. By controlling this value one can throttle
the pace at which containers are reclaimed from the cluster. After computing the total desired
preemption, the policy scales it back within this limit. Default value is `0.1` |
+| `yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity` | Maximum
amount of resources above the target capacity ignored for preemption. This defines a deadzone
around the target capacity that helps prevent thrashing and oscillations around the computed
target balance. High values would slow the time to capacity and (absent natural.completions)
it might prevent convergence to guaranteed capacity. Default value is  `0.1` |
+| `yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor` | Given a
computed preemption target, account for containers naturally expiring and preempt only this
percentage of the delta. This determines the rate of geometric convergence into the deadzone
(`MAX_IGNORED_OVER_CAPACITY`). For example, a termination factor of 0.5 will reclaim almost
95% of resources within 5 * #`WAIT_TIME_BEFORE_KILL`, even absent natural termination. Default
value is `0.2` |
+
+ The `CapacityScheduler` supports the following configurations in capacity-scheduler.xml
to control the preemption of application containers submitted to a queue.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.scheduler.capacity.<queue-path>.disable_preemption` | This configuration can
be set to `true` to selectively disable preemption of application containers submitted to
a given queue. This property applies only when system wide preemption is enabled by configuring
`yarn.resourcemanager.scheduler.monitor.enable` to *true* and `yarn.resourcemanager.scheduler.monitor.policies`
to *ProportionalCapacityPreemptionPolicy*. If this property is not set for a queue, then the
property value is inherited from the queue's parent. Default value is false.
+
+
 ###Other Properties
 
   * Resource Calculator


Mime
View raw message