hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashwin Shankar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1864) Fair Scheduler Dynamic Hierarchical User Queues
Date Tue, 06 May 2014 22:36:19 GMT

    [ https://issues.apache.org/jira/browse/YARN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13991261#comment-13991261
] 

Ashwin Shankar commented on YARN-1864:
--------------------------------------

Hey [~sandyr],
Agree to most of your comments,had some thoughts on couple.
bq. Do we need FSQueueType.PARENT_WITHOUT_CONFIGURED_LEAF as a separate queue type from PARENT?
I added that to reduce the extra worst case complexity of O(b^d) of iterating over all parent
queues(b=branching factor d = depth),
which could be big depending on the alloc configuration.
But I see your point,updateAllocationConfiguration happens rarely enough.I'm fine with removing
PARENT_WITHOUT_CONFIGURED_LEAF.

bq. Should be terminal if the nested rule is terminal?
I believe it should not be terminal for the following reason :
Even if create=true in the nested rule,the nested rule can return a configured leaf queue,which
will
cause the nestedUserQueue rule to skip to the next rule(due to the code below). So nestedUserQueue
rule is not terminal even 
if the nested rule is terminal.

{code}
if (configuredQueues.get(FSQueueType.LEAF).contains(queueName)) {
          return "";
        }
{code}


> Fair Scheduler Dynamic Hierarchical User Queues
> -----------------------------------------------
>
>                 Key: YARN-1864
>                 URL: https://issues.apache.org/jira/browse/YARN-1864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: scheduler
>            Reporter: Ashwin Shankar
>              Labels: scheduler
>         Attachments: YARN-1864-v1.txt, YARN-1864-v2.txt, YARN-1864-v3.txt, YARN-1864-v4.txt,
YARN-1864-v5.txt
>
>
> In Fair Scheduler, we want to be able to create user queues under any parent queue in
the hierarchy. For eg. Say user1 submits a job to a parent queue called root.allUserQueues,
we want be able to create a new queue called root.allUserQueues.user1 and run user1's job
in it.Any further jobs submitted by this user to root.allUserQueues will be run in this newly
created root.allUserQueues.user1.
> This is very similar to the 'user-as-default' feature in Fair Scheduler which creates
user queues under root queue. But we want the ability to create user queues under ANY parent
queue.
> Why do we want this ?
> 1. Preemption : these dynamically created user queues can preempt each other if its fair
share is not met. So there is fairness among users.
> User queues can also preempt other non-user leaf queue as well if below fair share.
> 2. Allocation to user queues : we want all the user queries(adhoc) to consume only a
fraction of resources in the shared cluster. By creating this feature,we could do that by
giving a fair share to the parent user queue which is then redistributed to all the dynamically
created user queues.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message