hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wan...@apache.org
Subject hadoop git commit: YARN-4826. Document configuration of ReservationSystem for CapacityScheduler. (Subru Krishnan via wangda)
Date Fri, 08 Apr 2016 23:34:04 GMT
Repository: hadoop
Updated Branches:
  refs/heads/branch-2.8 1b5b85e56 -> 67ee30272


YARN-4826. Document configuration of ReservationSystem for CapacityScheduler. (Subru Krishnan
via wangda)

(cherry picked from commit fcb3fcd4c674d58e5f40b17eff7048e8ae45d151)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md

(cherry picked from commit dac2021f870af674ab1a64e937c4f2a388dcc692)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/67ee3027
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/67ee3027
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/67ee3027

Branch: refs/heads/branch-2.8
Commit: 67ee3027238a9f306c0bc478222c22024dcd9b68
Parents: 1b5b85e
Author: Wangda Tan <wangda@apache.org>
Authored: Fri Apr 8 16:21:04 2016 -0700
Committer: Wangda Tan <wangda@apache.org>
Committed: Fri Apr 8 16:33:55 2016 -0700

----------------------------------------------------------------------
 .../src/site/markdown/CapacityScheduler.md      | 28 ++++++++++++++++++++
 1 file changed, 28 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/67ee3027/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 8c0b8c8..007842a 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -24,6 +24,7 @@ Hadoop: Capacity Scheduler
     * [Queue Properties](#Queue_Properties)
     * [Setup for application priority](#Setup_for_application_priority.)
     * [Capacity Scheduler container preemption](#Capacity_Scheduler_container_preemption)
+    * [Configuring `ReservationSystem` with `CapacityScheduler`](#Configuring_ReservationSystem_with_CapacityScheduler)
     * [Other Properties](#Other_Properties)
     * [Reviewing the configuration of the CapacityScheduler](#Reviewing_the_configuration_of_the_CapacityScheduler)
 * [Changing Queue Configuration](#Changing_Queue_Configuration)
@@ -241,6 +242,33 @@ The following configuration parameters can be configured in yarn-site.xml
to con
 | `yarn.scheduler.capacity.root.<queue>.acl_list_reservations` | The ACL which controls
who can *list* reservations to the given queue. If the given user/group has necessary ACLs
on the given queue they can list all applications. ACLs for this property *are not* inherited
from the parent queue if not specified. |
 | `yarn.scheduler.capacity.root.<queue>.acl_submit_reservations` | The ACL which controls
who can *submit* reservations to the given queue. If the given user/group has necessary ACLs
on the given queue they can submit reservations. ACLs for this property *are not* inherited
from the parent queue if not specified. |
 
+### Configuring `ReservationSystem` with `CapacityScheduler`
+
+ The `CapacityScheduler` supports the **ReservationSystem** which allows users to reserve
resources ahead of time. The application can request the reserved resources at runtime by
specifying the `reservationId` during submission. The following configuration parameters can
be configured in yarn-site.xml for `ReservationSystem`.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.reservation-system.enable` | *Mandatory* parameter: to enable the
`ReservationSystem` in the **ResourceManager**. Boolean value expected. The default value
is *false*, i.e. `ReservationSystem` is not enabled by default. |
+| `yarn.resourcemanager.reservation-system.class` | *Optional* parameter: the class name
of the `ReservationSystem`. The default value is picked based on the configured Scheduler,
i.e. if `CapacityScheduler` is configured, then it is `CapacityReservationSystem`. |
+| `yarn.resourcemanager.reservation-system.plan.follower` | *Optional* parameter: the class
name of the `PlanFollower` that runs on a timer, and synchronizes the `CapacityScheduler`
with the `Plan` and viceversa. The default value is picked based on the configured Scheduler,
i.e. if `CapacityScheduler` is configured, then it is `CapacitySchedulerPlanFollower`. |
+| `yarn.resourcemanager.reservation-system.planfollower.time-step` | *Optional* parameter:
the frequency in milliseconds of the `PlanFollower` timer. Long value expected. The default
value is *1000*. |
+
+
+The `ReservationSystem` is integrated with the `CapacityScheduler` queue hierachy and can
be configured for any **LeafQueue** currently. The `CapacityScheduler` supports the following
parameters to tune the `ReservationSystem`:
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.scheduler.capacity.<queue-path>.reservable` | *Mandatory* parameter: indicates
to the `ReservationSystem` that the queue's resources is available for users to reserve. Boolean
value expected. The default value is *false*, i.e. reservations are not enabled in *LeafQueues*
by default. |
+| `yarn.scheduler.capacity.<queue-path>.reservation-agent` | *Optional* parameter:
the class name that will be used to determine the implementation of the `ReservationAgent`
 which will attempt to place the user's reservation request in the `Plan`. The default value
is *org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy*.
|
+| `yarn.scheduler.capacity.<queue-path>.reservation-move-on-expiry` | *Optional* parameter
to specify to the `ReservationSystem` whether the applications should be moved or killed to
the parent reservable queue (configured above) when the associated reservation expires. Boolean
value expected. The default value is *true* indicating that the application will be moved
to the reservable queue. |
+| `yarn.scheduler.capacity.<queue-path>.show-reservations-as-queues` | *Optional* parameter
to show or hide the reservation queues in the Scheduler UI. Boolean value expected. The default
value is *false*, i.e. reservation queues will be hidden. |
+| `yarn.scheduler.capacity.<queue-path>.reservation-policy` | *Optional* parameter:
the class name that will be used to determine the implementation of the `SharingPolicy`  which
will validate if the new reservation doesn't violate any invariants.. The default value is
*org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityOverTimePolicy*. |
+| `yarn.scheduler.capacity.<queue-path>.reservation-window` | *Optional* parameter
representing the time in milliseconds for which the `SharingPolicy` will validate if the constraints
in the Plan are satisfied. Long value expected. The default value is one day. |
+| `yarn.scheduler.capacity.<queue-path>.instantaneous-max-capacity` | *Optional* parameter:
maximum capacity at any time in percentage (%) as a float that the `SharingPolicy` allows
a single user to reserve. The default value is 1, i.e. 100%. |
+| `yarn.scheduler.capacity.<queue-path>.average-capacity` | *Optional* parameter: the
average allowed capacity which will aggregated over the *ReservationWindow* in percentage
(%) as a float that the `SharingPolicy` allows a single user to reserve. The default value
is 1, i.e. 100%. |
+| `yarn.scheduler.capacity.<queue-path>.reservation-planner` | *Optional* parameter:
the class name that will be used to determine the implementation of the *Planner*  which will
be invoked if the `Plan` capacity fall below (due to scheduled maintenance or node failuers)
the user reserved resources. The default value is *org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.SimpleCapacityReplanner*
which scans the `Plan` and greedily removes reservations in reversed order of acceptance (LIFO)
till the reserved resources are within the `Plan` capacity |
+| `yarn.scheduler.capacity.<queue-path>.reservation-enforcement-window` | *Optional*
parameter representing the time in milliseconds for which the `Planner` will validate if the
constraints in the Plan are satisfied. Long value expected. The default value is one hour.
|
+
 ###Other Properties
 
   * Resource Calculator


Mime
View raw message