hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashwin Shankar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2395) FairScheduler: Preemption timeout should be configurable per queue
Date Tue, 26 Aug 2014 19:03:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14111148#comment-14111148
] 

Ashwin Shankar commented on YARN-2395:
--------------------------------------

[~kasha],
bq. If a parent queue is starved, wouldn't at least one of the child queues starve?
Not always. Here is an example :

Queue hierarchy :
root.lowPriorityLeaf - fair share = 10%
root.HighPriorityParent - fair share = 90% fairSharePreemptionThreshold=1
root.HighPriorityParent.child(1-10)
Scenario :
Apps running in root.lowPriorityLeaf, root.HighPriorityParent.child1, root.HighPriorityParent.child2(Remember
we now have fair share for active queues)
Following situation is possible :
root.lowPriorityLeaf : *usage = 55% demand = 55% fair share = 10%*
root.HighPriorityParent.child1:  *usage = 45% demand = 85% fair share = 45%*
root.HighPriorityParent.child2 : usage = 5% demand = 5% fair share = 45%

In above example, low priority queue with fair share 10% is taking up 55% of the cluster,
while HighPriorityParent.child1 needs 85%, but can get only 45% through preemption since thats
its fair share. Another point is HighPriorityParent.child2 has a fair share of 45%, but needs
only 5%.
*Note that both child1,child2 are NOT starved, but HighPriorityParent is starved.*

Use case is basically this : We want ALL 90% of the cluster resources to go to HighPriorityParent
whenever its needed by ANY of its children. We can do that by detecting starvation at parent
"HighPriorityParent" and preempt from lowPriorityLeaf.

> FairScheduler: Preemption timeout should be configurable per queue
> ------------------------------------------------------------------
>
>                 Key: YARN-2395
>                 URL: https://issues.apache.org/jira/browse/YARN-2395
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: fairscheduler
>            Reporter: Ashwin Shankar
>            Assignee: Wei Yan
>         Attachments: YARN-2395-1.patch, YARN-2395-2.patch, YARN-2395-3.patch, YARN-2395-3.patch
>
>
> Currently in fair scheduler, the preemption logic considers fair share starvation only
at leaf queue level. This jira is created to implement it at the parent queue as well.
> It involves :
> 1. Making "check for fair share starvation" and "amount of resource to preempt"  recursive
such that they traverse the queue hierarchy from root to leaf.
> 2. Currently fairSharePreemptionTimeout is a global config. We could make it configurable
on a per queue basis,so that we can specify different timeouts for parent queues.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message