hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
Date Tue, 05 Dec 2017 02:21:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277903#comment-16277903

Wangda Tan commented on YARN-7473:

Thanks [~suma.shivaprasad], I took a look at some of updated part (especially for the Configuration
passed to LeafQueue code paths). Some comments so far:

1) AbstractManagedParentQueue:
    CapacitySchedulerConfiguration leafQueueConfigs = new
        CapacitySchedulerConfiguration(new Configuration(false));
Should be 
    CapacitySchedulerConfiguration leafQueueConfigs = new
        CapacitySchedulerConfiguration(new Configuration(false), false);

//        .getConfiguration().getAllPropertiesByTag(YarnPropertyTag
      // .RESOURCEMANAGER).iterator()
Should be removed.

2) setEntitlement should be removed from ReservationQueue 

public void setEntitlement(String nodeLabel, QueueEntitlement entitlement)
This one should be removed. And see my next comment.

4) Why call updateCapacitiesToZero inside AutoCreatedLeafQueue#initialize? Will this cause
jittering of capacity? (When reinitialize happens, all queue’s capacities will be set to
0 and soon updated by CapacityManager?

5) Inside LeafQueue#setupQueueConfigs, now all methods are get from the Configuration passed-in.
However, I think we should only fetch queue-specific configs from the passed-in Configuration
reference. And for global configs such as 

nodeLocalityDelay = conf.getNodeLocalityDelay();
rackLocalityAdditionalDelay = conf.getRackLocalityAdditionalDelay();
rackLocalityFullReset = conf.getRackLocalityFullReset();

Should get from this.configuration.

We may need to add a test for this.

> Implement Framework and policy for capacity management of auto created queues 
> ------------------------------------------------------------------------------
>                 Key: YARN-7473
>                 URL: https://issues.apache.org/jira/browse/YARN-7473
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>            Reporter: Suma Shivaprasad
>            Assignee: Suma Shivaprasad
>         Attachments: YARN-7473.1.patch, YARN-7473.10.patch, YARN-7473.11.patch, YARN-7473.12.patch,
YARN-7473.2.patch, YARN-7473.3.patch, YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch,
YARN-7473.7.patch, YARN-7473.8.patch, YARN-7473.9.patch
> This jira mainly addresses the following
> 1.Support adding pluggable policies on parent queue for dynamically managing capacity/state
for leaf queues.
> 2. Implement  a default policy that manages capacity based on pending applications and
either grants guaranteed or zero capacity to queues based on parent's available guaranteed
> 3. Integrate with SchedulingEditPolicy framework to trigger this periodically and signal
scheduler to take necessary actions for capacity/queue management.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message