hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
Date Wed, 06 Dec 2017 17:59:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280585#comment-16280585
] 

Wangda Tan commented on YARN-7473:
----------------------------------

Thanks [~suma.shivaprasad] for updating the patch, several minor comments:

1) In AbstractCSQueue, following methods need to use csContext.getConf().

	  this.reservationsContinueLooking =
		  configuration.getReservationContinueLook();
   
   And following methods need to be updated:
   
   1.1 isQueueHierarchyPreemptionDisabled has two parts:
	   a. load from global config.
	   b. load from queue config.

	   The second part needs a Configuration passed in. 

   1.2 Similarly, getUserWeightsFromHierarchy should use the passed-in Configuration object.

2) In AutoCreatedLeafQueue, following statements:

{code}
      //update queue usage before setting capacity to 0
      CSQueueUtils.updateQueueStatistics(resourceCalculator, clusterResource,
          this, labelManager, null);
{code}

Should after updateCapacitiesToZero. Since queue's capacity is updated to zero. You can move
it into updateCapacitiesToZero as well. We need to make sure used capacity properly calculated
for every capacity changes. 

Similarly, In

{code}
public void setEntitlement(String nodeLabel, QueueEntitlement entitlement)
{code}

We should call (Please note that we only update usedCapacities for given {{nodeLabel}})

{code}
CSQueueUtils.updateQueueStatistics(resourceCalculator, clusterResource,
          this, labelManager, nodeLabel); 
{code}

And I found we should remove the implementation of {{setEntitlement(QueueEntitlement entitlement)}}.
It should call setEntitlement(CommonNodeLabelsManager.NO_LABEL, entitlement)} instead.

3) There's a potential NPE in CapacityScheduler.java
{code}
      if (queue == null || !(AbstractAutoCreatedLeafQueue.class
          .isAssignableFrom(queue.getClass()) )) {
        throw new SchedulerDynamicEditException(
            "Queue " + queue.getQueueName() + " is not an implementation of "
                + "AbstractAutoCreatedLeafQueue");
      }
{code}

It should be 

{code}
      if (queue == null) {
         // LOG and exception
      } else if (!(AbstractAutoCreatedLeafQueue.class
          .isAssignableFrom(queue.getClass())) {
         // LOG and exception
      }
{code}

> Implement Framework and policy for capacity management of auto created queues 
> ------------------------------------------------------------------------------
>
>                 Key: YARN-7473
>                 URL: https://issues.apache.org/jira/browse/YARN-7473
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>            Reporter: Suma Shivaprasad
>            Assignee: Suma Shivaprasad
>         Attachments: YARN-7473.1.patch, YARN-7473.10.patch, YARN-7473.11.patch, YARN-7473.12.patch,
YARN-7473.12.patch, YARN-7473.13.patch, YARN-7473.14.patch, YARN-7473.2.patch, YARN-7473.3.patch,
YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch, YARN-7473.8.patch,
YARN-7473.9.patch
>
>
> This jira mainly addresses the following
>  
> 1.Support adding pluggable policies on parent queue for dynamically managing capacity/state
for leaf queues.
> 2. Implement  a default policy that manages capacity based on pending applications and
either grants guaranteed or zero capacity to queues based on parent's available guaranteed
capacity.
> 3. Integrate with SchedulingEditPolicy framework to trigger this periodically and signal
scheduler to take necessary actions for capacity/queue management.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message