hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sandy Ryza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1961) Fair scheduler preemption doesn't work for non-leaf queues
Date Sat, 03 May 2014 01:29:15 GMT

    [ https://issues.apache.org/jira/browse/YARN-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988518#comment-13988518

Sandy Ryza commented on YARN-1961:

Hey Ashwin,
The history is only that we haven't yet added support for that property in parent queues.
 I agree that it would be a helpful thing to add.

> Fair scheduler preemption doesn't work for non-leaf queues
> ----------------------------------------------------------
>                 Key: YARN-1961
>                 URL: https://issues.apache.org/jira/browse/YARN-1961
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: scheduler
>    Affects Versions: 2.4.0
>            Reporter: Ashwin Shankar
>              Labels: scheduler
> Setting minResources and minSharePreemptionTimeout to a non-leaf queue doesn't cause
preemption to happen when that non-leaf queue is below minResources and there are outstanding
demands in that non-leaf queue.
> Here is an example fs allocation config(partial) :
> {code:xml}
> <queue name="abc">
>   <minResources>3072 mb,0 vcores</minResources>
>   <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
>     <queue name="childabc1">
>     </queue>
>     <queue name="childabc2">
>     </queue>
>  </queue>
>  {code}
> With the above configs,preemption doesn't seem to happen if queue abc is below minShare
and it has outstanding unsatisfied demands from apps in its child queues. Ideally in such
cases we would like preemption to kick off and reclaim resources from other queues(not under
queue abc).
> Looking at the code it seems like preemption checks for starvation only at the leaf queue
level and not at the parent level.
> {code:title=FairScheduler.java|borderStyle=solid}
> boolean isStarvedForMinShare(FSLeafQueue sched)
> boolean isStarvedForFairShare(FSLeafQueue sched)
> {code}
> This affects our use case where we have a parent queue with probably a 100 unconfigured
leaf queues under it.We want to give a minshare to the parent queue to protect all the leaf
queues under it,but we cannot do it due to this bug.

This message was sent by Atlassian JIRA

View raw message