hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashwin Shankar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues
Date Fri, 25 Apr 2014 00:53:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980569#comment-13980569

Ashwin Shankar commented on YARN-1864:

bq. Can we use configuredQueues for this? We shouldn't be touching the QueueManager from inside
the QueuePlacementPolicy.

Yes we can use configuredQueues for this(instead of queueMgr code) like the following,but
there is a case we will miss :
 if (configuredQueues.get(FSQueueType.LEAF).contains(nestedRuleOutput)) {
            return "";
Here is the problem with above code. There are five types of queues the nested rule can return
1. Existing Configured Parent Queue.
2. Existing Configured Leaf Queue.
3. Existing Unconfigured Parent Queue(created when create=true)
4. Existing Unconfigured Leaf Queue(created when create=true)
5. Non existent queue.

Using above code snippet handles all the cases correctly except case4.
When we get an existing unconfigured leaf queue from nested rule,we will not skip the rule
and would result in an error later.
Using queueMgr would cover all 5 cases.

Having said that,case-4 occurs in corner cases,like when the admin makes a regular rule into
a nested rule
without bouncing RM. For eg : Initially he might have had primaryGroup with create=true as
a regular rule.
This could create some unconfigured leaf queues. Then he nests this primaryGroup under userQueueBelow
Now those unconfigured leaf queues can again be returned by the nested rule, and will not
be caught by the code snippet 
mentioned above and user would get an error later. But using queueMgr would cause a skip and
hopefully would run the app in
some other queue.

Let me know if you want to support this ? If not, configuresQueues will work,I'll change that
in the next patch.
Also were there any other comments you had in mind which you've not mentioned yet ?

> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>                 Key: YARN-1864
>                 URL: https://issues.apache.org/jira/browse/YARN-1864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Ashwin Shankar
>              Labels: scheduler
>         Attachments: YARN-1864-v1.txt, YARN-1864-v2.txt, YARN-1864-v3.txt
> In Fair Scheduler, we want to be able to create user queues under any parent queue in
the hierarchy. For eg. Say user1 submits a job to a parent queue called root.allUserQueues,
we want be able to create a new queue called root.allUserQueues.user1 and run user1's job
in it.Any further jobs submitted by this user to root.allUserQueues will be run in this newly
created root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which creates
user queues under root queue. But we want the ability to create user queues under ANY parent
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other if its fair
share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume only a
fraction of resources in the shared cluster. By creating this feature,we could do that by
giving a fair share to the parent user queue which is then redistributed to all the dynamically
created user queues.

This message was sent by Atlassian JIRA

View raw message