hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1707) Making the CapacityScheduler more dynamic
Date Tue, 26 Aug 2014 23:05:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111520#comment-14111520
] 

Carlo Curino commented on YARN-1707:
------------------------------------

Wangda,

Thanks for the great feedback. You spot a bunch of oddities that were there due to previous
versions of the reservation system, 
but not needed anymore, I think the updated version is definitely cleaner. 

We address 1,2 by:
 * moving addQueue and removeQueue to PlanQueue (as they were only invoked on instances of
the subclass).
 * making uniform checks from within the PlanQueue for capacity > 0, and throw uniform
SchedulerConfigEditException
 * fixing the log, and making the logs more uniform

We address 3,6 by:
 * merge addCapacity and subtractCapacity into a single changeCapacity
 * make checks of range limits 0,1 
(this reduced code both in CS and ReservationQueue... good call!)

We address 4 by:
 * getReservableQueues() has been renamed to getPlanQueues()

Regarding 5: ReservationQueue#getQueueName 
 * This is the result of our previous conversations with Vinod, Bikas, and Arun. The idea
is that the user should not be 
aware of the fact that we use queues to implement reservations, and thus it shouldn't see
the name of the reservation 
queue to be listed in the UI, but rather the name of the parent PlanQueue. More precisely,
we have options for the UI 
to show or not the subqueues, but this differentiation is needed here to allow that: getQueueName
for a ReservationQueue
return the parent, while getReservationQueueName() returns the actual local name.

Regarding 7: DynamicQueueConf
* We currently are only dynamically assigning capacity, but you can imagine in the future
that this is extended to set many
more parameters for a queue (user-limit factors, max applications, etc..). The conf-based
mechanism is future-proofing against this.

Regarding 8:  ParentQueue#setChildQueues
* I don't understand the comment. This check is automatically bypassed for PlanQueue (that
by design have no children see 
CapacityScheduler near line 562).

We are testing the new version of the patch now, and will post patch soon.


> Making the CapacityScheduler more dynamic
> -----------------------------------------
>
>                 Key: YARN-1707
>                 URL: https://issues.apache.org/jira/browse/YARN-1707
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>              Labels: capacity-scheduler
>         Attachments: YARN-1707.2.patch, YARN-1707.3.patch, YARN-1707.patch
>
>
> The CapacityScheduler is a rather static at the moment, and refreshqueue provides a rather
heavy-handed way to reconfigure it. Moving towards long-running services (tracked in YARN-896)
and to enable more advanced admission control and resource parcelling we need to make the
CapacityScheduler more dynamic. This is instrumental to the umbrella jira YARN-1051.
> Concretely this require the following changes:
> * create queues dynamically
> * destroy queues dynamically
> * dynamically change queue parameters (e.g., capacity) 
> * modify refreshqueue validation to enforce sum(child.getCapacity())<= 100% instead
of ==100%
> We limit this to LeafQueues. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message