hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Welch (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2496) [YARN-796] Changes for capacity scheduler to support allocate resource respect labels
Date Fri, 19 Sep 2014 21:33:34 GMT

    [ https://issues.apache.org/jira/browse/YARN-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141342#comment-14141342
] 

Craig Welch commented on YARN-2496:
-----------------------------------

Let's say user UserU is running two jobs, JobA and JobB, in queue QueueAB, which has both
LabelA and LabelB.  JobA has label LabelA, JobB has LabelB.  It's a two node cluster, Node1
and Node2.  Node1 has LabelA, Node2 has LabelB

Let's say the user has access to 100% of the cluster (just the one queue, etc).

Let's say that JobA is using 4G of Ram and JobB is also using 4G.  Let's say each node has
6G.

Headroom Calculation for JobA:

userConsumed = 8G
maxCapacityConsiderLabelA = 6G (Node1 only)
headroom = -2G (assume it will normalize to 0G)

However, the user should still be able to use the remaining 2G for JobA, as they are only
using 4 of the 6G available to that label. 

The issue I see is userConsumed, as maxCapacityConsiderLabel considers the label, but userConsumed
does not, it should only be "userConsumedForLabel", if it is, JobA would see 2G as it should
(as the consumed for LabelA is only 4G.)  The problem, I think, is in subtracting a cross-label
value from a per-label value.

> [YARN-796] Changes for capacity scheduler to support allocate resource respect labels
> -------------------------------------------------------------------------------------
>
>                 Key: YARN-2496
>                 URL: https://issues.apache.org/jira/browse/YARN-2496
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-2496.patch, YARN-2496.patch, YARN-2496.patch, YARN-2496.patch
>
>
> This JIRA Includes:
> - Add/parse labels option to {{capacity-scheduler.xml}} similar to other options of queue
like capacity/maximum-capacity, etc.
> - Include a "default-label-expression" option in queue config, if an app doesn't specify
label-expression, "default-label-expression" of queue will be used.
> - Check if labels can be accessed by the queue when submit an app with labels-expression
to queue or update ResourceRequest with label-expression
> - Check labels on NM when trying to allocate ResourceRequest on the NM with label-expression
> - Respect  labels when calculate headroom/user-limit



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message