hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From su...@apache.org
Subject hadoop git commit: YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru).
Date Wed, 25 Oct 2017 22:07:58 GMT
Repository: hadoop
Updated Branches:
  refs/heads/trunk 7da15eac7 -> 3fae67538


YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru).


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/3fae6753
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/3fae6753
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/3fae6753

Branch: refs/heads/trunk
Commit: 3fae675383489129b3ca3c66683a1215d0c6edf0
Parents: 7da15ea
Author: Subru Krishnan <subru@apache.org>
Authored: Wed Oct 25 15:07:50 2017 -0700
Committer: Subru Krishnan <subru@apache.org>
Committed: Wed Oct 25 15:07:50 2017 -0700

----------------------------------------------------------------------
 .../reservation/CapacityOverTimePolicy.java     |  3 ---
 .../src/site/markdown/FairScheduler.md          | 24 ++++++++++++++++++++
 .../src/site/markdown/ReservationSystem.md      |  2 +-
 3 files changed, 25 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fae6753/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
index acd5774..1f3f9bc 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
@@ -56,9 +56,6 @@ public class CapacityOverTimePolicy extends NoOverCommitPolicy {
   private float maxInst;
   private float maxAvg;
 
-  // For now this is CapacityScheduler specific, but given a hierarchy in the
-  // configuration structure of the schedulers (e.g., SchedulerConfiguration)
-  // it should be easy to remove this limitation
   @Override
   public void init(String reservationQueuePath,
       ReservationSchedulerConfiguration conf) {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fae6753/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
index 64c450b..8d53e57 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
@@ -109,6 +109,8 @@ The allocation file must be in XML format. The format contains five types
of ele
 
     * **allowPreemptionFrom**: determines whether the scheduler is allowed to preempt resources
from the queue. The default is true. If a queue has this property set to false, this property
will apply recursively to all child queues.
 
+    * **reservation**: indicates to the `ReservationSystem` that the queue's resources is
available for users to reserve. This only applies for leaf queues. A leaf queue is not reservable
if this property isn't configured.
+
 * **User elements**: which represent settings governing the behavior of individual users.
They can contain a single property: maxRunningApps, a limit on the number of running apps
for a particular user.
 
 * **A userMaxAppsDefault element**: which sets the default running app limit for any users
whose limit is not otherwise specified.
@@ -127,6 +129,12 @@ The allocation file must be in XML format. The format contains five types
of ele
 
 * **A defaultQueueSchedulingPolicy element**: which sets the default scheduling policy for
queues; overridden by the schedulingPolicy element in each queue if specified. Defaults to
"fair".
 
+* **A reservation-agent element**: which sets the class name of the implementation of the
`ReservationAgent`, which attempts to place the user's reservation request in the `Plan`.
The default value is `org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy`.
+
+* **A reservation-policy element**: which sets the class name of the implementation of the
`SharingPolicy`, which validates if the new reservation doesn't violate any invariants. The
default value is `org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityOverTimePolicy`.
+
+* **A reservation-planner element**: which sets the class name of the implementation of the
`Planner`, which is invoked if the `Plan` capacity fall below (due to scheduled maintenance
or node failuers) the user reserved resources. The default value is `org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.SimpleCapacityReplanner`,
which scans the `Plan` and greedily removes reservations in reversed order of acceptance (LIFO)
till the reserved resources are within the `Plan` capacity.
+
 * **A queuePlacementPolicy element**: which contains a list of rule elements that tell the
scheduler how to place incoming apps into queues. Rules are applied in the order that they
are listed. Rules may take arguments. All rules accept the "create" argument, which indicates
whether the rule can create a new queue. "Create" defaults to true; if set to false and the
rule would place the app in a queue that is not configured in the allocations file, we continue
on to the next rule. The last rule must be one that can never issue a continue. Valid rules
are:
 
     * **specified**: the app is placed into the queue it requested. If the app requested
no queue, i.e. it specified "default", we continue. If the app requested a queue name starting
or ending with period, i.e. names like ".q1" or "q1." will be rejected.
@@ -159,6 +167,9 @@ The allocation file must be in XML format. The format contains five types
of ele
       <aclSubmitApps>charlie</aclSubmitApps>
       <minResources>5000 mb,0vcores</minResources>
     </queue>
+    <queue name="sample_reservable_queue">
+      <reservation></reservation>
+    </queue>
   </queue>
 
   <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
@@ -201,6 +212,19 @@ The root queue's ACLs are "\*" by default which, because ACLs are passed
down, m
 
 Reservation Access Control Lists (ACLs) allow administrators to control who may take reservation
actions on particular queues. They are configured with the aclAdministerReservations, aclListReservations,
and the aclSubmitReservations properties, which can be set per queue. Currently the supported
administrative actions are updating and deleting reservations. An administrator may also submit
and list *all* reservations on the queue. These properties take values in a format like "user1,user2
group1,group2" or " group1,group2". Actions on a queue are permitted if the user/group is
a member of the reservation ACL. Note that any user can update, delete, or list their own
reservations. If reservation ACLs are enabled but not defined, everyone will have access.
 
+### Configuring `ReservationSystem`
+
+The Fair Scheduler supports the **ReservationSystem** which allows users to reserve resources
ahead of time. The application can request reserved resources at runtime by specifying the
`reservationId` during submission. The following configuration parameters can be configured
in yarn-site.xml for `ReservationSystem`.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.reservation-system.enable` | *Mandatory* parameter: to enable the
`ReservationSystem` in the **ResourceManager**. Boolean value expected. The default value
is *false*, i.e. `ReservationSystem` is not enabled by default. |
+| `yarn.resourcemanager.reservation-system.class` | *Optional* parameter: the class name
of the `ReservationSystem`. The default value is picked based on the configured Scheduler,
i.e. if `FairScheduler` is configured, then it is `FairReservationSystem`. |
+| `yarn.resourcemanager.reservation-system.plan.follower` | *Optional* parameter: the class
name of the `PlanFollower` that runs on a timer, and synchronizes the `FairScheduler` with
the `Plan` and vice versa. The default value is picked based on the configured Scheduler,
i.e. if `FairScheduler` is configured, then it is `FairSchedulerPlanFollower`. |
+| `yarn.resourcemanager.reservation-system.planfollower.time-step` | *Optional* parameter:
the frequency in milliseconds of the `PlanFollower` timer. Long value expected. The default
value is *1000*. |
+
+The `ReservationSystem` is integrated with the Fair Scheduler queue hierarchy and can be
configured for and only for leaf queues. The detailed instructions are in section `Allocation
file format`.
+
 ##Administration
 
 The fair scheduler provides support for administration at runtime through a few mechanisms:

http://git-wip-us.apache.org/repos/asf/hadoop/blob/3fae6753/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
----------------------------------------------------------------------
diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
index cd4ae54..ac3269d 100644
--- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
+++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
@@ -60,4 +60,4 @@ With reference to the figure above, a typical reservation proceeds as follows:
 Configuring the Reservation System
 ----------------------------------
 
-Configuring the `ReservationSystem` is simple. Currently we have added support for *reservations*
in both `CapacityScheduler` and `FairScheduler`. You can mark any **leaf queue** in the **capacity-scheduler.xml**
or **fair-scheduler.xml** as available for "reservations" (see [CapacityScheduler](./CapacityScheduler.html#Configuring_ReservationSystem_with_CapacityScheduler)
and the [FairScheduler](./FairScheduler.html) for details). Then the capacity/fair share within
that queue can be used for making reservations. Jobs can still be submitted to the *reservable
queue* without a reservation, in which case they will be run in best-effort mode in whatever
capacity is left over by the jobs running within active reservations.
+Configuring the `ReservationSystem` is simple. Currently we have added support for *reservations*
in both `CapacityScheduler` and `FairScheduler`. You can mark any **leaf queue** in the **capacity-scheduler.xml**
or **fair-scheduler.xml** as available for "reservations" (see [CapacityScheduler](./CapacityScheduler.html#Configuring_ReservationSystem_with_CapacityScheduler)
and the [FairScheduler](./FairScheduler.html#Configuring_ReservationSystem) for details).
Then the capacity/fair share within that queue can be used for making reservations. Jobs can
still be submitted to the *reservable queue* without a reservation, in which case they will
be run in best-effort mode in whatever capacity is left over by the jobs running within active
reservations.


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-commits-help@hadoop.apache.org


Mime
View raw message