hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1241) In Fair Scheduler maxRunningApps does not work for non-leaf queues
Date Wed, 09 Oct 2013 21:28:43 GMT

    [ https://issues.apache.org/jira/browse/YARN-1241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13790879#comment-13790879

Sandy Ryza commented on YARN-1241:

Thanks for the review Karthik. Will post a rebased patch on latest trunk with your suggestions
except where discussed below.

bq. Should we make this a synchronized method along with other accessors of runnable and non-runnable
appScheds? In case, we make the scheduler multi-threaded?
I think you're right that this will ultimately need to be synchronized when we remove the
super-coarse synchronization that we have currently. But, as a bunch of methods in similar
situations currently aren't synchronized, I'd rather do it in a separate JIRA where we can
take this on all at once.

bq. Making a mental note. The assumption here is that we don't use demand to determine fairness
between schedulables. Correct?

bq. Should we even consider the demand for non-runnable schedulables?
My thinking I think it's more useful for a cluster operator to know about the total amount
of resources demanded by all jobs submitted to the cluster.  E.g. someone might look at a
queue's demand because they want to know whether anybody in it is waiting for anything, and
leaving out the non-runnable apps would mask this from them.

bq. MaxRunningAppsEnforcer should be singleton?
As there's only a single FairScheduler per RM, there will only end up being single instance.
If for some reason we wanted to run multiple FairSchedulers in a process (for testing maybe),
we would need a MaxRunningAppsEnforcer for each.

bq. I am assuming the ListMultimap choice is to maintain the ordering and give preference
to the earliest submitted job within a queue. Wondering if it would make sense to use any
other metric than submit time for "improved fairness"?
Right.  Definitely sounds conceivable to me that we might want to order them by something
else in the future, but I don't we need to make the data structure more general until then.

bq. IIUC, we are trying to check if the completing of this app makes any other applications
runnable now? Shouldn't we break out of the loop as soon as we encounter a queue with zero
non-runnable apps?
Yeah that's what we are trying to check.  childqueueX might have no pending apps itself, but
if a queue higher up in the hierarchy parentqueueY has a maxRunningApps set, an app completion
in childqueueX could allow an app in some other distant child of parentqueueY to become runnable.
 I'll update the comment to try to explain a little more of what's being done.

bq. Also, would be nice to have separate unit tests for MaxRunningAppsEnforcer.
You're right.  I will add a couple in.

> In Fair Scheduler maxRunningApps does not work for non-leaf queues
> ------------------------------------------------------------------
>                 Key: YARN-1241
>                 URL: https://issues.apache.org/jira/browse/YARN-1241
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.1.0-beta
>            Reporter: Sandy Ryza
>            Assignee: Sandy Ryza
>         Attachments: YARN-1241-1.patch, YARN-1241-2.patch, YARN-1241-3.patch, YARN-1241.patch
> Setting the maxRunningApps property on a parent queue should make it that the sum of
apps in all subqueues can't exceed it

This message was sent by Atlassian JIRA

View raw message