hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7117) Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
Date Tue, 29 Aug 2017 00:18:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144580#comment-16144580
] 

Wangda Tan commented on YARN-7117:
----------------------------------

Discussed with [~clayb]/[~sunilg]l/[~vinodkv] offline (Thanks Clay for sharing internal use
cases). Here is our initial proposal to get more thoughts: 
- Parent queue can be marked to allow auto creation of leaf queues. 
(such as {{prefix.<queue-path>.auto-queue-creation.enabled}}, default is off). We allow
no sub queue specified for such parent queue.
- Minimum resource could be specified for queues which are automatically created. (such as

{{prefix.<queue-path>.auto-queue-creation.subqueue-minimum-resource}}). After YARN-5881,
absolute resources can be specified for auto created queues.
- CS treats automatically created queues no different from normal queues, which means scheduler
will use existing logic to do preemption / fairness allocation / queue-ordering / user-limit,
etc. for auto-created queue. 
- ACL of created queue should be determined by policy. For example, if we expect create different
queue for different user, admin may set 
{{prefix.<queue-path>.auto-queue-creation.admin-acl-policy=user-name-equals-to-queue-name}}.
- Auto-create queue flag can be specified in queue-mapping policy, default is off.

A related issue (maybe it's better to discuss on a separate JIRA) is: it's possible that queues
are created but not actively used, so we could allow guaranteed resources are overcommitted.
(For example a parent queue with 100G guaranteed memory, and there're 200 sub queues created
under the parent, each queue has 1G guaranteed memory, but most of the sub queues are not
being used).

To solve the issue, scheduler may need to maintain a list of {{#active-leaf-queues}} under
one parent (An active-leaf-queue means a leaf queue has at least one app not in final state).
Parent queue's guaranteed resource will be checked and enforced when state of leaf queue's
changed to active. Application submission will be rejected if 
- {{Σ(leafQueue.guaranteed) (leafQueue ∈ \{active-leaf-queues\}) > parent.guaranteed)}}.


> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
> ----------------------------------------------------------------------------------
>
>                 Key: YARN-7117
>                 URL: https://issues.apache.org/jira/browse/YARN-7117
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>
> Currently Capacity Scheduler doesn't support auto creation of queues when doing queue
mapping. We saw more and more use cases which has complex queue mapping policies configured
to handle application to queues mapping. 
> The most common use case of CapacityScheduler queue mapping is to create one queue for
each user/group. However update {{capacity-scheduler.xml}} and {{RMAdmin:refreshQueues}} needs
to be done when new user/group onboard. One of the option to solve the problem is automatically
create queues when new user/group arrives.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message