hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2154) FairScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
Date Fri, 05 Sep 2014 18:54:28 GMT

    [ https://issues.apache.org/jira/browse/YARN-2154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123380#comment-14123380
] 

Karthik Kambatla commented on YARN-2154:
----------------------------------------

At an abstract level, I propose the following:
# For each queue, the child {{Schedulable}}s should be sorted in the ascending order of (desiredShare
- fairShare). In case of FSLeafQueue, this corresponds to running applications.
# Add {{FSQueue#estimateStarvation(List<FSAppAttempt>)}} to find enough applications,
considering minshare and fairshare starvation in that order. Or, should we have two methods,
one for each kind of starvation. 
# For a starved queue, we do a depth-first traversal (in the order of deprivation) to find
these applications. If none of child queues/applications are under their shares, we pick the
application (if it exists) with a positive demand and least over its share.
# Once the application list is computed, we traverse through the ResourceRequests (ordered
first by application, second by priority) and preempt any matching containers currently assigned
to an application over its share.
# As a follow up, we can consider looking at this application list first at schedule time.

Thoughts? 

> FairScheduler: Improve preemption to preempt only those containers that would satisfy
the incoming request
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-2154
>                 URL: https://issues.apache.org/jira/browse/YARN-2154
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.4.0
>            Reporter: Karthik Kambatla
>            Assignee: Karthik Kambatla
>            Priority: Critical
>
> Today, FairScheduler uses a spray-gun approach to preemption. Instead, it should only
preempt resources that would satisfy the incoming request. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message