hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1707) Making the CapacityScheduler more dynamic
Date Fri, 29 Aug 2014 22:53:54 GMT

    [ https://issues.apache.org/jira/browse/YARN-1707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14115999#comment-14115999
] 

Jian He commented on YARN-1707:
-------------------------------

Hi Carlo, thanks for your work !  I looked at the patch, some comments and questions:
- to simplify, we can use getNumApplications() method
{code}
disposableLeafQueue.getApplications().size() > 0
        || disposableLeafQueue.pendingApplications.size() > 0
{code}
- PlanQueue.java 80 column limit
- why “newQueue.changeCapacity(sesConf.getCapacity());” is inside the check and “queue.setMaxCapacity(sesConf.getMaxCapacity());”
is outside the check
- CapacityScheduler#getReservationQueueNames seems getting the child reservation queues of
the given plan queue. We can use the planQueue#childQueues directly
- DynamicQueueConf, how about calling it QueueEntitlement to be consistent ?
- CapacityScheduler#parseQueue method, I think we can simplify the condition for isReservableQueue
flag something like this:
 {code}
    boolean isReservableQueue = conf.isReservableQueue(fullQueueName);
    if (isReservableQueue) {
      ParentQueue parentQueue = 
          new PlanQueue(csContext, queueName, parent,
              oldQueues.get(queueName));
      queue = hook.hook(parentQueue);
    } else if ((childQueueNames == null || childQueueNames.length == 0))
{code}
- just to simplify, this log msg may be put after previous “qiter.remove();” to avoid
the removed boolean flag.
{code}
    if (LOG.isDebugEnabled()) {
      LOG.debug("updateChildQueues (action: remove queue): " + removed + " "
          + getChildQueuesToPrint());
    }
{code}
- we can add a new reinitialize in ReservationQueue which does all these initializations.

{code}
      CSQueueUtils.updateQueueStatistics(
          schedulerContext.getResourceCalculator(), ses, this,
          schedulerContext.getClusterResource(),
          schedulerContext.getMinimumResourceCapability());
      ses.reinitialize(ses, clusterResource);
      ((ReservationQueue) ses).setMaxApplications(this
          .getMaxApplicationsForReservations());
      ((ReservationQueue) ses).setMaxApplicationsPerUser(this
          .getMaxApplicationsPerUserForReservation());
{code}
- IIUC, right now, queueName here is for the planQueue(inherits parentQueue), and the reservationID
is for the reservationQueue(inherits from leafQueue). I think if we can get the proper reservationQueueName(leafQueue)
upfront and pass it as the queueName parameter into this method, we can avoid some if/else
condition changes inside this method and the method signature. 
{code}
  private synchronized void addApplication(ApplicationId applicationId,
    String queueName, String user, boolean isAppRecovering,
    ReservationId reservationID)
{code}

> Making the CapacityScheduler more dynamic
> -----------------------------------------
>
>                 Key: YARN-1707
>                 URL: https://issues.apache.org/jira/browse/YARN-1707
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacityscheduler
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>              Labels: capacity-scheduler
>         Attachments: YARN-1707.2.patch, YARN-1707.3.patch, YARN-1707.4.patch, YARN-1707.patch
>
>
> The CapacityScheduler is a rather static at the moment, and refreshqueue provides a rather
heavy-handed way to reconfigure it. Moving towards long-running services (tracked in YARN-896)
and to enable more advanced admission control and resource parcelling we need to make the
CapacityScheduler more dynamic. This is instrumental to the umbrella jira YARN-1051.
> Concretely this require the following changes:
> * create queues dynamically
> * destroy queues dynamically
> * dynamically change queue parameters (e.g., capacity) 
> * modify refreshqueue validation to enforce sum(child.getCapacity())<= 100% instead
of ==100%
> We limit this to LeafQueues. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message