hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7117) Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
Date Wed, 30 Aug 2017 20:57:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16148040#comment-16148040

Carlo Curino commented on YARN-7117:

I have very skimmed the thread very briefly, but it appears to me that you are looking at
lots of the problems are similar to those solved by the {{PlanQueue}} {{ReservationQueue}}
and {{PlanFollower}}. 
There we dealt with dynamic creation of queues, adding up to 100% as well as we can provide
(thanks to the planning aspects) stronger guarantees for the Oozie like use cases (recurring
aspects of this in YARN-5326).  

I would ask you to look at that stuff, and figure out whether you can leverage some/most of
it, instead of building a parallel solution. 
Overall the {{PlanFollower}} design of "observe an external signal and publish it to scheduler"
seems clean and powerful, as it decouples the CS inner workings from outside (slower evolving)

> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
> ----------------------------------------------------------------------------------
>                 Key: YARN-7117
>                 URL: https://issues.apache.org/jira/browse/YARN-7117
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> Currently Capacity Scheduler doesn't support auto creation of queues when doing queue
mapping. We saw more and more use cases which has complex queue mapping policies configured
to handle application to queues mapping. 
> The most common use case of CapacityScheduler queue mapping is to create one queue for
each user/group. However update {{capacity-scheduler.xml}} and {{RMAdmin:refreshQueues}} needs
to be done when new user/group onboard. One of the option to solve the problem is automatically
create queues when new user/group arrives.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message