hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
Date Tue, 05 Dec 2017 02:21:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277903#comment-16277903
] 

Wangda Tan commented on YARN-7473:
----------------------------------

Thanks [~suma.shivaprasad], I took a look at some of updated part (especially for the Configuration
passed to LeafQueue code paths). Some comments so far:

1) AbstractManagedParentQueue:
{code}
    CapacitySchedulerConfiguration leafQueueConfigs = new
        CapacitySchedulerConfiguration(new Configuration(false));
{code} 
Should be 
{code}
    CapacitySchedulerConfiguration leafQueueConfigs = new
        CapacitySchedulerConfiguration(new Configuration(false), false);
{code} 

And 
{code}
//        .getConfiguration().getAllPropertiesByTag(YarnPropertyTag
      // .RESOURCEMANAGER).iterator()
{code}
Should be removed.

2) setEntitlement should be removed from ReservationQueue 

3) 
{code}
public void setEntitlement(String nodeLabel, QueueEntitlement entitlement)
{code}
This one should be removed. And see my next comment.

4) Why call updateCapacitiesToZero inside AutoCreatedLeafQueue#initialize? Will this cause
jittering of capacity? (When reinitialize happens, all queue’s capacities will be set to
0 and soon updated by CapacityManager?

5) Inside LeafQueue#setupQueueConfigs, now all methods are get from the Configuration passed-in.
However, I think we should only fetch queue-specific configs from the passed-in Configuration
reference. And for global configs such as 

{code}
nodeLocalityDelay = conf.getNodeLocalityDelay();
rackLocalityAdditionalDelay = conf.getRackLocalityAdditionalDelay();
rackLocalityFullReset = conf.getRackLocalityFullReset();
{code}

Should get from this.configuration.

We may need to add a test for this.

> Implement Framework and policy for capacity management of auto created queues 
> ------------------------------------------------------------------------------
>
>                 Key: YARN-7473
>                 URL: https://issues.apache.org/jira/browse/YARN-7473
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>            Reporter: Suma Shivaprasad
>            Assignee: Suma Shivaprasad
>         Attachments: YARN-7473.1.patch, YARN-7473.10.patch, YARN-7473.11.patch, YARN-7473.12.patch,
YARN-7473.2.patch, YARN-7473.3.patch, YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch,
YARN-7473.7.patch, YARN-7473.8.patch, YARN-7473.9.patch
>
>
> This jira mainly addresses the following
>  
> 1.Support adding pluggable policies on parent queue for dynamically managing capacity/state
for leaf queues.
> 2. Implement  a default policy that manages capacity based on pending applications and
either grants guaranteed or zero capacity to queues based on parent's available guaranteed
capacity.
> 3. Integrate with SchedulingEditPolicy framework to trigger this periodically and signal
scheduler to take necessary actions for capacity/queue management.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message