hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1921) Allow to override queue prefix, where new queues will be created
Date Thu, 10 Apr 2014 21:58:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13965913#comment-13965913

Hadoop QA commented on YARN-1921:

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3548//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3548//console

This message is automatically generated.

> Allow to override queue prefix, where new queues will be created
> ----------------------------------------------------------------
>                 Key: YARN-1921
>                 URL: https://issues.apache.org/jira/browse/YARN-1921
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 2.3.0
>         Environment: Yarn 2.3.0
>            Reporter: Andrey Stepachev
>         Attachments: YARN-1921.patch, YARN-1921.patch
> Fair scheduler has a couple of QueuePlacementRules. Those rules can create queues, if
they not exists with hardcoded prefix "root.". 
> Consider an example: we have a placement rule, which creates user's queue if it not exists.
Current implementation creates it at "root." prefix  Suppose that this user runs a big job.
In that case it will get a fair share of resources because queue will be created at 'root.'
with default settings, and that affects all other users of the cluster. 
> Of course, FairScheduler can place such users to default queue, but in that case if user
submits a big queue it will eat resources of whole queue, and we know that no preemption can
be done within one queue (Or i'm wrong?). So effectively one user can usurp all default queue
> To solve that I created a patch, which allows to override "root." prefix in QueuePlacementRules.
Thats gives us flexibility to automatically create queues for users or group of users under
predefined queue. So, every user will get a separate queue and will share parent queue resources
and can't usurp all resources, because parent node can be configured to preempt tasks.
> Consider example (parent queue specified for each rule):
> {code:title=policy.xml|borderStyle=solid}
> <queuePlacementPolicy>
>   <rule name='specified' parent='granted'/>
>   <rule name='user'  parent='guests'/>
> </queuePlacementPolicy>
> {code}
> With such definition queue requirements will give us:
> {code:title=Example.java|borderStyle=solid}
> "root.granted.specifiedq" == policy.assignAppToQueue("specifiedq", "someuser");
> "root.guests.someuser" == policy.assignAppToQueue("default", "someuser");
> "root.guests.otheruser" == policy.assignAppToQueue("default", "otheruser"); 
> {code}

This message was sent by Atlassian JIRA

View raw message