hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7117) Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
Date Tue, 29 Aug 2017 17:36:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145746#comment-16145746

Wangda Tan commented on YARN-7117:

Thanks [~asuresh] / [~jlowe],

Both of you commented on unused queues. IIRC, in fair scheduler, there's no guaranteed resource
concept. minResource affect how queues sharing resource, but it is not a guaranteed value
since it allows 
- {{Σ(child.minResource) > parent.minResource.}}

I'm agree what [~jlowe] mentioned:
bq. In short, if we're going to allow new, previously illegal configurations for auto-queues
then arguably we should be consistent and allow them elsewhere as well.

To make behavior of automatically created queue consistent with manual created queues, it
might be problematic if we allow overcommit of auto created queue but not allow overcommit
of manual queues. 
Adding queue with zero guaranteed resource is one possible solution, but it may not be good
enough for resource sharing since many users don't frequently use cluster.
Instead, maybe admin can configure a "delete-policy" which delete queues which are unused
for X secs. Thoughts? 

> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
> ----------------------------------------------------------------------------------
>                 Key: YARN-7117
>                 URL: https://issues.apache.org/jira/browse/YARN-7117
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> Currently Capacity Scheduler doesn't support auto creation of queues when doing queue
mapping. We saw more and more use cases which has complex queue mapping policies configured
to handle application to queues mapping. 
> The most common use case of CapacityScheduler queue mapping is to create one queue for
each user/group. However update {{capacity-scheduler.xml}} and {{RMAdmin:refreshQueues}} needs
to be done when new user/group onboard. One of the option to solve the problem is automatically
create queues when new user/group arrives.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message