hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1961) Fair scheduler preemption doesn't work for non-leaf queues
Date Wed, 02 Mar 2016 01:35:18 GMT

     [ https://issues.apache.org/jira/browse/YARN-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Karthik Kambatla updated YARN-1961:
-----------------------------------
    Issue Type: Sub-task  (was: Bug)
        Parent: YARN-4752

> Fair scheduler preemption doesn't work for non-leaf queues
> ----------------------------------------------------------
>
>                 Key: YARN-1961
>                 URL: https://issues.apache.org/jira/browse/YARN-1961
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler, scheduler
>    Affects Versions: 2.4.0
>            Reporter: Ashwin Shankar
>            Assignee: Ashwin Shankar
>              Labels: scheduler
>
> Setting minResources and minSharePreemptionTimeout to a non-leaf queue doesn't cause
preemption to happen when that non-leaf queue is below minResources and there are outstanding
demands in that non-leaf queue.
> Here is an example fs allocation config(partial) :
> {code:xml}
> <queue name="abc">
>   <minResources>3072 mb,0 vcores</minResources>
>   <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
>     <queue name="childabc1">
>     </queue>
>     <queue name="childabc2">
>     </queue>
>  </queue>
>  {code}
> With the above configs,preemption doesn't seem to happen if queue abc is below minShare
and it has outstanding unsatisfied demands from apps in its child queues. Ideally in such
cases we would like preemption to kick off and reclaim resources from other queues(not under
queue abc).
> Looking at the code it seems like preemption checks for starvation only at the leaf queue
level and not at the parent level.
> {code:title=FairScheduler.java|borderStyle=solid}
> boolean isStarvedForMinShare(FSLeafQueue sched)
> boolean isStarvedForFairShare(FSLeafQueue sched)
> {code}
> This affects our use case where we have a parent queue with probably a 100 unconfigured
leaf queues under it.We want to give a minshare to the parent queue to protect all the leaf
queues under it,but we cannot do it due to this bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message