hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
Date Thu, 28 Jan 2016 04:20:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120725#comment-15120725

Naganarasimha G R commented on YARN-3215:

Thanks [~sunilg] & [~wangda] for sharing your views.
bq.  Applications could ask nothing in the first AllocateRequest, get headroom back and decide
how many resources to ask, we cannot presume application's behavior. Telling an app: "your
headroom is 0" but there're plenty of resources looks not good enough to me.
IIUC i have taken this approach in {{CapacityHeadRoomProvider.getHeadroom}}
 if (requestedPartitions.isEmpty() || (requestedPartitions.size() == 1
        && requestedPartitions.contains(RMNodeLabelsManager.NO_LABEL))) {
           headroom = queue.getHeadroom(user, queueCurrentLimit, clusterResource,
So basically when there is no partitions requested i send the headroom of default Partition
hence IMO it doesnt break the compatability now. 
Would this not be sufficient ? i can add a test case for the same and verify it !

> Respect labels in CapacityScheduler when computing headroom
> -----------------------------------------------------------
>                 Key: YARN-3215
>                 URL: https://issues.apache.org/jira/browse/YARN-3215
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3215.v1.001.patch, YARN-3215.v2.001.patch, YARN-3215.v2.002.patch
> In existing CapacityScheduler, when computing headroom of an application, it will only
consider "non-labeled" nodes of this application.
> But it is possible the application is asking for labeled resources, so headroom-by-label
(like 5G resource available under node-label=red) is required to get better resource allocation
and avoid deadlocks such as MAPREDUCE-5928.
> This JIRA could involve both API changes (such as adding a label-to-available-resource
map in AllocateResponse) and also internal changes in CapacityScheduler.

This message was sent by Atlassian JIRA

View raw message