hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7117) Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
Date Wed, 30 Aug 2017 13:33:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16147241#comment-16147241

Jason Lowe commented on YARN-7117:

bq. This is a valid concern, instead of deleting queue, how about stopping the queue?

Probably a bit better depending upon the use case.  Still may be surprising to a user who
is submitting regularly (e.g.: via Oozie) and suddenly their submissions fail because the
queue is stopped due to some other users suddenly submitting some jobs at an inopportune time
and forcing this user's queue to stop as a result.  I guess that's not a valid use case for
this.  As long as it's clear how it behaves and that matches expectations of users looking
to use this feature I guess it's OK.

It may be even better to just leave them as-is and require someone to cleanup unused queues
in order to allow more auto-queues, otherwise I'm not sure we could ever deploy this.  It
would allow one user to remove capabilities previously given to another user which is a denial
of service.  In that sense it seems better to be up front and either say, "no auto queue for
you until someone frees up space" or "hey, here's a queue with little-to-no guarantees but
it could have a lot of capacity depending upon other user activity."  That latter model (as
a single queue for us but could be a bunch of separate, per-user queues if necessary) has
worked pretty well for us for ad-hoc jobs.

> Capacity Scheduler: Support Auto Creation of Leaf Queues While Doing Queue Mapping
> ----------------------------------------------------------------------------------
>                 Key: YARN-7117
>                 URL: https://issues.apache.org/jira/browse/YARN-7117
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
> Currently Capacity Scheduler doesn't support auto creation of queues when doing queue
mapping. We saw more and more use cases which has complex queue mapping policies configured
to handle application to queues mapping. 
> The most common use case of CapacityScheduler queue mapping is to create one queue for
each user/group. However update {{capacity-scheduler.xml}} and {{RMAdmin:refreshQueues}} needs
to be done when new user/group onboard. One of the option to solve the problem is automatically
create queues when new user/group arrives.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message