hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashwin Shankar (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues
Date Thu, 24 Apr 2014 05:19:16 GMT

     [ https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ashwin Shankar updated YARN-1864:

    Attachment: YARN-1864-v3.txt

[~sandyr] Hey Sandy,Patch refreshed with most of your comments.
bq. Any reason why it can't?
I removed this code in the latest patch.
I put that previously since it didn't make sense to me why one would want to create 'user
queues under a user queue'. But I'm fine not having this restriction.

bq. The QueuePlacementPolicy isn't in responsible for verifying this. We expect to hit an
error later on if the queue returned by the QueuePlacementPolicy isn't a leaf queue. This
can happen with other rules, for example with the "specified" rule if someone tries to explicitly
submit to a parent queue.
Yes makes sense,but here is a use case which I'm trying to support where user doesn't specify
a queue.
*Use case* : Nested rule is also a regular rule
<rule name="userQueueBelow">
       <rule name="primaryGroup" create="false">
<rule name="primaryGroup" create="false">
If we want some primary group queues to have user queues underneath and some primary groups
to be plain leaf queues,we could
create a queue placement policy like above. In the 'userQueueBelow' rule,if we don't check
if the nested rule returned a leaf queue
and skip,we will never reach the 3rd rule and always throw an error later when user's primary
group is a leaf queue.

bq. Along similar lines to the previous comment, I don't think we need this logic, or isNestedQueue
and FSQueueType, in the queue placement policy. If the rule places an app in "root.engineering.ashwin"
and root.engineering happens to be a leaf or root.ashwin happens to be a parent, we'll end
up throwing an error later, which is about the best we can do. Let me know if there are cases
I'm missing or if I'm not thinking this through deeply enough.
Removed isNested logic in latest patch. But FSQueueType is needed to get configured parent
queues and configured leaf queues.
Currently in hadoop trunk code, 'configuredQueues' in QueuePlacementRule.java is a Set containing
only configured leaf queues. 
However with userQueueBelow rule,the nested rule can return configured parent queues when
create is false.

bq. This has gotten to be a fairly big patch - mind making these changed to the "default"
rule in a separate JIRA?
Sure,I've removed changes to default rule in the patch,I'll create a new jira for this.

bq.The name "HierarchicalUserQueue" seems a kind of weird to me.
Yes I agree. I think your suggestion 'UserQueueBelow' seems reasonable.I've changed the name
at all places in the latest patch.

bq. I've got a few stylistic comments in addition to these.
Sure please let me know,I'll incorporate it in my next patch.

> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>                 Key: YARN-1864
>                 URL: https://issues.apache.org/jira/browse/YARN-1864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Ashwin Shankar
>              Labels: scheduler
>         Attachments: YARN-1864-v1.txt, YARN-1864-v2.txt, YARN-1864-v3.txt
> In Fair Scheduler, we want to be able to create user queues under any parent queue in
the hierarchy. For eg. Say user1 submits a job to a parent queue called root.allUserQueues,
we want be able to create a new queue called root.allUserQueues.user1 and run user1's job
in it.Any further jobs submitted by this user to root.allUserQueues will be run in this newly
created root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which creates
user queues under root queue. But we want the ability to create user queues under ANY parent
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other if its fair
share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume only a
fraction of resources in the shared cluster. By creating this feature,we could do that by
giving a fair share to the parent user queue which is then redistributed to all the dynamically
created user queues.

This message was sent by Atlassian JIRA

View raw message