Return-Path: 
 <yarn-issues-return-26168-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 75433100D5
	for <apmail-hadoop-yarn-issues-archive@minotaur.apache.org>;
 Thu, 10 Apr 2014 21:28:30 +0000 (UTC)
Received: (qmail 34136 invoked by uid 500); 10 Apr 2014 21:28:25 -0000
Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org
Received: (qmail 33849 invoked by uid 500); 10 Apr 2014 21:28:20 -0000
Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:yarn-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:yarn-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:yarn-issues@hadoop.apache.org>
List-Id: <yarn-issues.hadoop.apache.org>
Reply-To: yarn-issues@hadoop.apache.org
Delivered-To: mailing list yarn-issues@hadoop.apache.org
Received: (qmail 33728 invoked by uid 99); 10 Apr 2014 21:28:16 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 10 Apr 2014 21:28:16 +0000
Date: Thu, 10 Apr 2014 21:28:16 +0000 (UTC)
From: "Andrey Stepachev (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12707735.1397146469162.96268.1397165296311@arcas>
In-Reply-To: <JIRA.12707735.1397146469162@arcas>
References: <JIRA.12707735.1397146469162@arcas>
Subject: [jira] [Updated] (YARN-1921) Allow to override queue prefix, where
 new queues will be created
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/YARN-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrey Stepachev updated YARN-1921:
-----------------------------------

    Attachment: YARN-1921.patch

> Allow to override queue prefix, where new queues will be created
> ----------------------------------------------------------------
>
>                 Key: YARN-1921
>                 URL: https://issues.apache.org/jira/browse/YARN-1921
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: scheduler
>    Affects Versions: 2.3.0
>         Environment: Yarn 2.3.0
>            Reporter: Andrey Stepachev
>         Attachments: YARN-1921.patch, YARN-1921.patch
>
>
> Fair scheduler has a couple of QueuePlacementRules. Those rules can create queues, if they not exists with hardcoded prefix "root.". 
> Consider an example: we have a placement rule, which creates user's queue if it not exists. Current implementation creates it at "root." prefix  Suppose that this user runs a big job. In that case it will get a fair share of resources because queue will be created at 'root.' with default settings, and that affects all other users of the cluster. 
> Of course, FairScheduler can place such users to default queue, but in that case if user submits a big queue it will eat resources of whole queue, and we know that no preemption can be done within one queue (Or i'm wrong?). So effectively one user can usurp all default queue resources.
> To solve that I created a patch, which allows to override "root." prefix in QueuePlacementRules. Thats gives us flexibility to automatically create queues for users or group of users under predefined queue. So, every user will get a separate queue and will share parent queue resources and can't usurp all resources, because parent node can be configured to preempt tasks.
> Consider example (parent queue specified for each rule):
> {code:title=policy.xml|borderStyle=solid}
> <queuePlacementPolicy>
>   <rule name='specified' parent='granted'/>
>   <rule name='user'  parent='guests'/>
> </queuePlacementPolicy>
> {code}
> With such definition queue requirements will give us:
> {code:title=Example.java|borderStyle=solid}
> "root.granted.specifiedq" == policy.assignAppToQueue("specifiedq", "someuser");
> "root.guests.someuser" == policy.assignAppToQueue("default", "someuser");
> "root.guests.otheruser" == policy.assignAppToQueue("default", "otheruser"); 
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)