hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3215) Respect labels in CapacityScheduler when computing headroom
Date Wed, 13 Jan 2016 16:22:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15096466#comment-15096466

Sunil G commented on YARN-3215:

Thanks [~Naganarasimha Garla] for pushing this ticket. I had one doubt here.
+  Resource getQueueMaxResource(String nodePartition, Resource clusterResource) {
+    return Resources.multiplyAndNormalizeDown(resourceCalculator,
+        labelManager.getResourceByLabel(nodePartition, clusterResource),
+        queueCapacities.getAbsoluteMaximumCapacity(nodePartition), minimumAllocation);
+  }
{{getAbsoluteMaximumCapacity}} is used here. In LeafQueue#getHeadroom, {{currentPartitionResourceLimit}}
is now derived from this max capacity of a queue (per label).
So could this be a bigger value considering same label may present in multiple queues. Thus
by projecting more headroom, is there chance that more requests may be pushed by different
applications in different queues (but same label). Pls correct me if I am wrong.

> Respect labels in CapacityScheduler when computing headroom
> -----------------------------------------------------------
>                 Key: YARN-3215
>                 URL: https://issues.apache.org/jira/browse/YARN-3215
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Wangda Tan
>            Assignee: Naganarasimha G R
>         Attachments: YARN-3215.v1.001.patch
> In existing CapacityScheduler, when computing headroom of an application, it will only
consider "non-labeled" nodes of this application.
> But it is possible the application is asking for labeled resources, so headroom-by-label
(like 5G resource available under node-label=red) is required to get better resource allocation
and avoid deadlocks such as MAPREDUCE-5928.
> This JIRA could involve both API changes (such as adding a label-to-available-resource
map in AllocateResponse) and also internal changes in CapacityScheduler.

This message was sent by Atlassian JIRA

View raw message