hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashwin Shankar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues
Date Mon, 14 Apr 2014 04:55:18 GMT

     [ https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ashwin Shankar updated YARN-1864:

    Attachment: YARN-1864-v2.txt

Patch YARN-1864-v2.txt has the following features :
1. Hierarchical User queue implemented using nested rules.
With this feature, we can create user queues with primary group,secondary group,default,specified
rules etc by just nesting that rule.
Please see TestQueuePlacementPolicy.java in the patch for examples/use cases. It is extensible,so
any new rule in the future can get user queue support with hierarchicalUserQueue rule without
much code changes.
2. A way to configure parent queues in the fair scheduler alloc xml
without specifying any leaf queues through a new attribute called 'type'
in the queue element. 
<queue name="queue1" type="parent">
//other queue properties
Please see TestFairScheduler#testHierarchicalUserQueue for an example.

3. 'default' queue placement rule can take a default queue as input.
If no default queue specified, we fall back to root.default queue.
<rule name='default' queue='root.queue1'/>

I've added a bunch of unit tests and also manually tested hierarchicalUserQueue rule with
combinations of other rules. Also verified that preemption involving hierarchical user queues

> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>                 Key: YARN-1864
>                 URL: https://issues.apache.org/jira/browse/YARN-1864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Ashwin Shankar
>              Labels: scheduler
>         Attachments: YARN-1864-v1.txt, YARN-1864-v2.txt
> In Fair Scheduler, we want to be able to create user queues under any parent queue in
the hierarchy. For eg. Say user1 submits a job to a parent queue called root.allUserQueues,
we want be able to create a new queue called root.allUserQueues.user1 and run user1's job
in it.Any further jobs submitted by this user to root.allUserQueues will be run in this newly
created root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which creates
user queues under root queue. But we want the ability to create user queues under ANY parent
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other if its fair
share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume only a
fraction of resources in the shared cluster. By creating this feature,we could do that by
giving a fair share to the parent user queue which is then redistributed to all the dynamically
created user queues.

This message was sent by Atlassian JIRA

View raw message