hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suma Shivaprasad (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
Date Mon, 04 Dec 2017 14:05:01 GMT

     [ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Suma Shivaprasad updated YARN-7473:
    Attachment: YARN-7473.11.patch

Thanks [~sunilg] and [~wangda] Attaching patch with most of the review comments addressed

{quote} Once queue capacity is set to 0, same queue will be assigned back with some capacity
based on available capacity. If capacity is not there, queue will be still 0. If many such
queues with 0 capacity is starving, queue which got submitted with an app first will be selected.
We might need to consider priority also here. {quote}
   Discussed with [~wangda] and [~sunilg] offline. App priority is intra queue and will not
be usable as a criteria while considering to choose leaf queues 
{quote} AbstractManagedParentQueue#validateQueueEntitlementChange is directly operating on
capacity. When absolute resource will get merged, this code will be a problem? {quote} 
   This will require changes in policy class and could be addressed in a separate jira once
Aboslute resource is merged.
In initializeLimitsFromTemplate, should below code to be setMaxApplications(leafQueueTemplate.getMaxApps());
related to parent queue as well. What if some one provided more apps in template which could
violate parent max-apps? 
{quote} In validateConfigurations, does 0 a valid capacity? one could configure 0 as capacity
and +ve integer for max-capacity?{quote}
Yes, 0 is a valid configuration for queue capacity 

How to configure orderingPolicy for AutoCreatedLeafQueue?
in below code, better to avoid _ for queue config names
Can be configured through <parent-queue-prefix>.leaf-queue-template.ordering-policy.

1684	  public static final String QUEUE_MANAGEMENT_MONITORING_INTERVAL =
1685	      QUEUE_MANAGEMENT_CONFIG_PREFIX + "monitoring_interval";
In CapacitySchedulerContext, better to use MonotonicClock

In LeafQueue,
2011	  public void setMaxAMResourcePerQueuePercent(
2012	      float maxAMResourcePerQueuePercent) {
2013	    this.maxAMResourcePerQueuePercent = maxAMResourcePerQueuePercent;
2014	  }
how are we handling node labels?
  Discussed with [~wangda] and [~sunilg] offline. This has been addressed through changes
for using CapacitySchedulerConfiguration instead of individual configs for a ManagedParentQueue

PendingApplicationComparator could reuse existing fifo/fair app comparators?
  Discussed with [~wangda] and [~sunilg] offline. fifo/fair app comparators cant be reused
since they are mostly for intra queue app ordering and we cannot consider priority etc while
ordering leaf queues within a parent queue as mentioned above.

Also this patch reverts changes done for using a common AutoCreatedLeafQueue for both reservations
and auto created queues. This was needed since the initialize/reinitialize for ReservationQueues
and AutoCreatedLeafQueue are no longer common.

> Implement Framework and policy for capacity management of auto created queues 
> ------------------------------------------------------------------------------
>                 Key: YARN-7473
>                 URL: https://issues.apache.org/jira/browse/YARN-7473
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>            Reporter: Suma Shivaprasad
>            Assignee: Suma Shivaprasad
>         Attachments: YARN-7473.1.patch, YARN-7473.10.patch, YARN-7473.11.patch, YARN-7473.2.patch,
YARN-7473.3.patch, YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch,
YARN-7473.8.patch, YARN-7473.9.patch
> This jira mainly addresses the following
> 1.Support adding pluggable policies on parent queue for dynamically managing capacity/state
for leaf queues.
> 2. Implement  a default policy that manages capacity based on pending applications and
either grants guaranteed or zero capacity to queues based on parent's available guaranteed
> 3. Integrate with SchedulingEditPolicy framework to trigger this periodically and signal
scheduler to take necessary actions for capacity/queue management.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message