hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "MENG DING (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
Date Fri, 04 Sep 2015 16:54:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731048#comment-14731048
] 

MENG DING commented on YARN-4108:
---------------------------------

Hi, [~leftnoteasy]

I also feel that the logic you proposed is a good staring point overall. Just want to confirm
that I understand it correctly. For pending asks with hard locality requirements, I think
this logic works the best. However, for other pending asks, are we able to achieve optimal
preemption (i.e. sufficiently preemptable resources with the lowest cost of preemption as
per [~jlowe])? For example, just because {{node.available + preemptable > application.next_request}}
doesn't necessarily mean that the preemption cost is the lowest on this node. Maybe we need
to have a combination of reservation continuous looking + delayed scheduling mechanism to
ensure that we have done calculation of preemption cost on enough hosts for the pending ask.
But then I feel this approach might be too expensive ...

> CapacityScheduler: Improve preemption to preempt only those containers that would satisfy
the incoming request
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption is more
effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality (I only
want to use rack-1) / node-constraints (YARN-3409) / black-list (I don't want to use rack1
and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), cross applicaiton
preemption (such as priority-based (YARN-1963) / fairness-based (YARN-3319)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message