hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1241) In Fair Scheduler maxRunningApps does not work for non-leaf queues
Date Wed, 09 Oct 2013 20:50:45 GMT

    [ https://issues.apache.org/jira/browse/YARN-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790846#comment-13790846

Karthik Kambatla commented on YARN-1241:

Good to see we are separating out runnable and non-runnable apps - keen to see how much this
improves the schedule-time overhead. Comments below: 

+  private final List<AppSchedulable> appScheds = // apps that are runnable
How about renaming this field to runnableAppScheds?

+  public void makeAppRunnable(AppSchedulable appSched) {
Should we make this a synchronized method along with other accessors of runnable and non-runnable
appScheds? In case, we make the scheduler multi-threaded? 

      if (Resources.equals(demand, maxRes)) {
Making a mental note. The assumption here is that we don't use demand to determine fairness
between schedulables. Correct?

    for (AppSchedulable sched : nonRunnableAppScheds) {
      if (Resources.equals(demand, maxRes)) {
      updateDemandForApp(sched, maxRes);
Should we even consider the demand for non-runnable schedulables? Would it make sense to ignore
the non-runnable schedulables for all book-keeping until they become runnable? 

In FairScheduler, the PriorityQueue import looks spurious.

MaxRunningAppsEnforcer should be singleton?

  private ListMultimap<String, AppSchedulable> usersNonRunnableApps;
I am assuming the ListMultimap choice is to maintain the ordering and give preference to the
earliest submitted job within a queue. Wondering if it would make sense to use any other metric
than submit time for "improved fairness"?

   * Runs in O(n log(n)) where n is the number of queues under that are under
   * the highest queue that went from having no slack to having slack.
The comment should have a single under?

    // Update runnable app bookkeeping for queues
    // Find the queue highest in the tree which the app removal means something
    // new can be run
    FSQueue highestQueueAppsNowRunnable =
        (queue.getNumRunnableApps() == queueMgr.getQueueMaxApps(queue.getName()) - 1)
        ? queue : null;
    FSParentQueue parent = queue.getParent();
    while (parent != null) {
      if (parent.getNumRunnableApps()
          == queueMgr.getQueueMaxApps(parent.getName())) {
        highestQueueAppsNowRunnable = parent;
      parent = parent.getParent();
The comment is a little confusing. Rename highestQueueAppsNowRunnable to highestQueueWithAppsNowRunnable?
Also, I am a little confused here. IIUC, we are trying to check if the completing of this
app makes any other applications runnable now? Shouldn't we break out of the loop as soon
as we encounter a queue for with zero non-runnable apps? Also, would be nice to have separate
unit tests for MaxRunningAppsEnforcer.

> In Fair Scheduler maxRunningApps does not work for non-leaf queues
> ------------------------------------------------------------------
>                 Key: YARN-1241
>                 URL: https://issues.apache.org/jira/browse/YARN-1241
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.1.0-beta
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: YARN-1241-1.patch, YARN-1241-2.patch, YARN-1241-3.patch, YARN-1241.patch
> Setting the maxRunningApps property on a parent queue should make it that the sum of
apps in all subqueues can't exceed it

This message was sent by Atlassian JIRA

View raw message