hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-3124) Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
Date Wed, 11 Feb 2015 00:22:11 GMT

     [ https://issues.apache.org/jira/browse/YARN-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Wangda Tan updated YARN-3124:
    Attachment: YARN-3124.3.patch

bq. Merge CapacitySchedulerConfiguration#setCapacitiesByLabels and CSQueueUtils#setAbsoluteCapacitiesByNodeLabels
into a single method

bq. CapacitySchedulerConfiguration#normalizeAccessibleNodeLabels - should AbstractCSQueue#accessibleLabels
be updated as well ?
No, if we have ANY in accessible node labels, and cluster's node label collection could change,
so we need to keep the "ANY" in case any changes of clusterNodeLabels

bq. why union? newCapacities.getExistingNodeLabels is enough ?
No, this method's semantic is, replace all (absolute)(maximum)capacities, so if we only use
newCapacity.existingNodeLabels, labels which are not existed in new "ExistingNodeLabels" will
not be replaced.

bq. Can the existing get*CapacityByLabel can be removed? use queueCapacities#get*capacity

bq. null for the queueCapacity ? then we can remove the parameter
Updated setQueueConfig logic, see below

bq. remove this?

bq. CSQueueUtils.setAbsoluteCapacitiesByNodeLabel may be inside AbstractCSQueue
Now I consolidate all capacities updating fields to CSQueueUtils.loadUpdateAndCheckCapacities

bq. QueueCapacities#getExistingNodeLabels - > getNodeLabels?
It seems getExistingNodeLabels is more expressive to me :)

bq. why CSQueueUtils.setAbsoluteCapacitiesByNodeLabels(queueCapacities, parent); has to be
called in ReservationQueue#reinitialize
I added a grand comment in ReservationQueue to explain why we do this

And in beyond, I found we have ParentQueue/LeafQueue/AbstractCSQueue.setupQueueConfig, there're
lots of parameters we need to maintain, it's not simple when we want to add any new (configurable)
field to queues, I basically removed all parameters in setupQueueConfig. Instead, all (configurable)
fields will be read and initialized from configuration.

Even if we read the Configuration object twice, but I think it doesn't affect performance
while reinitializing, and thus we can get simpler structure of queue initialization.

*Attached ver.3 patch*

> Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
> ------------------------------------------------------------------------------------------------
>                 Key: YARN-3124
>                 URL: https://issues.apache.org/jira/browse/YARN-3124
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: api, client, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-3124.1.patch, YARN-3124.2.patch, YARN-3124.3.patch
> After YARN-3098, capacities-by-label (include used-capacity/maximum-capacity/absolute-maximum-capacity,
etc.) should be tracked in QueueCapacities.
> This patch is targeting to make capacities-by-label in CS Queues are all tracked by QueueCapacities.

This message was sent by Atlassian JIRA

View raw message