hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashwin Shankar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues
Date Thu, 10 Apr 2014 23:53:16 GMT

    [ https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13966024#comment-13966024

Ashwin Shankar commented on YARN-1864:

Hi [~octo47],
We want to be able to support user queues for any rule(now and in the future) without needing
to add code 
in that rule. I had a discussion with Sandy last week and we felt implementing hierarchicalUserQueues
using nested rules would make
it a) extensible and b) clean. For eg if we want user queues for Primary group,all we have
to do is :
<rule = hierarhicalUserQueue>
      <rule=primaryGroup />

The nested rule would be applied first and based on what it returns we can make a decision
at the HUQ level
whether to put it in a user queue underneath or skip to the next rule.
By this way we can also have 'create' flag at both hierarchicalUserQueue level and the nest
rule level which gives
the admin granularity of control over creating new queues.
If someone writes any other rule in future and want user queue support,they just have to nest
it with HUQ rule and things would just work.
No extra attributes in the xml is needed in the new rule.

As part of this patch, another thing I'm writing is to be able to mention parent queues without
leaf queues in the alloc xml,
which can then be used as user queues. I'm done with code,will write tests and post a patch
with all these features this week.

> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>                 Key: YARN-1864
>                 URL: https://issues.apache.org/jira/browse/YARN-1864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Ashwin Shankar
>              Labels: scheduler
>         Attachments: YARN-1864-v1.txt
> In Fair Scheduler, we want to be able to create user queues under any parent queue in
the hierarchy. For eg. Say user1 submits a job to a parent queue called root.allUserQueues,
we want be able to create a new queue called root.allUserQueues.user1 and run user1's job
in it.Any further jobs submitted by this user to root.allUserQueues will be run in this newly
created root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which creates
user queues under root queue. But we want the ability to create user queues under ANY parent
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other if its fair
share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume only a
fraction of resources in the shared cluster. By creating this feature,we could do that by
giving a fair share to the parent user queue which is then redistributed to all the dynamically
created user queues.

This message was sent by Atlassian JIRA

View raw message