hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues
Date Wed, 26 Mar 2014 17:17:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948162#comment-13948162

Sandy Ryza commented on YARN-1864:

Hey Ashwin,
Thanks for working on this.  This is something I've thought we'd need for a while.

I'd like to see this work with queue placement policies in the sense that one should not always
need to explicitly specify the parent queue when using this feature.  The queue placement
policy should be able to handle both choosing the parent queue and the user queue underneath.
 I realize this isn't super straightforward to add under the queue placement policy design.
 A couple ideas that I haven't fully thought through yet:
* Based on the "hierarchicalUserQueue" queue option, place the job in a user queue after the
queue placement policy assigns a parent queue.
* Have a "userQueueUnderQueue" queue placement rule that has another queue placement rule
nested inside of it.  If the inner queue placement rule matches the input and returns a queue,
place the job in a user queue underneath that queue.

If we can get the config file syntax to look decent, the latter seems somewhat more elegant
to me architecturally, because all the queue placement stuff is concentrated in one place.

> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>                 Key: YARN-1864
>                 URL: https://issues.apache.org/jira/browse/YARN-1864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Ashwin Shankar
>              Labels: scheduler
>             Fix For: 2.4.0
>         Attachments: YARN-1864-v1.txt
> In Fair Scheduler, we want to be able to create user queues under any parent queue in
the hierarchy. For eg. Say user1 submits a job to a parent queue called root.allUserQueues,
we want be able to create a new queue called root.allUserQueues.user1 and run user1's job
in it.Any further jobs submitted by this user to root.allUserQueues will be run in this newly
created root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which creates
user queues under root queue. But we want the ability to create user queues under ANY parent
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other if its fair
share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume only a
fraction of resources in the shared cluster. By creating this feature,we could do that by
giving a fair share to the parent user queue which is then redistributed to all the dynamically
created user queues.

This message was sent by Atlassian JIRA

View raw message