Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Fri, 19 Sep 2014 21:33:34 +0000 (UTC)
From: "Craig Welch (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12738747.1409734526000.81839.1411162414565@Atlassian.JIRA>
In-Reply-To: <JIRA.12738747.1409734526000@Atlassian.JIRA>
References: <JIRA.12738747.1409734526000@Atlassian.JIRA>
 <JIRA.12738747.1409734526952@arcas>
Subject: [jira] [Commented] (YARN-2496) [YARN-796] Changes for capacity
 scheduler to support allocate resource respect labels
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141342#comment-14141342 ] 

Craig Welch commented on YARN-2496:
-----------------------------------

Let's say user UserU is running two jobs, JobA and JobB, in queue QueueAB, which has both LabelA and LabelB.  JobA has label LabelA, JobB has LabelB.  It's a two node cluster, Node1 and Node2.  Node1 has LabelA, Node2 has LabelB

Let's say the user has access to 100% of the cluster (just the one queue, etc).

Let's say that JobA is using 4G of Ram and JobB is also using 4G.  Let's say each node has 6G.

Headroom Calculation for JobA:

userConsumed = 8G
maxCapacityConsiderLabelA = 6G (Node1 only)
headroom = -2G (assume it will normalize to 0G)

However, the user should still be able to use the remaining 2G for JobA, as they are only using 4 of the 6G available to that label. 

The issue I see is userConsumed, as maxCapacityConsiderLabel considers the label, but userConsumed does not, it should only be "userConsumedForLabel", if it is, JobA would see 2G as it should (as the consumed for LabelA is only 4G.)  The problem, I think, is in subtracting a cross-label value from a per-label value.

> [YARN-796] Changes for capacity scheduler to support allocate resource respect labels
> -------------------------------------------------------------------------------------
>
>                 Key: YARN-2496
>                 URL: https://issues.apache.org/jira/browse/YARN-2496
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
>
>
> This JIRA Includes:
> - Add/parse labels option to {{capacity-scheduler.xml}} similar to other options of queue like capacity/maximum-capacity, etc.
> - Include a "default-label-expression" option in queue config, if an app doesn't specify label-expression, "default-label-expression" of queue will be used.
> - Check if labels can be accessed by the queue when submit an app with labels-expression to queue or update ResourceRequest with label-expression
> - Check labels on NM when trying to allocate ResourceRequest on the NM with label-expression
> - Respect  labels when calculate headroom/user-limit


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)