hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
Date Tue, 01 Dec 2015 18:51:11 GMT

    [ https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034346#comment-15034346
] 

Wangda Tan commented on YARN-4108:
----------------------------------

Uploading design doc with initial POC patch for review.

I put general ideas to design doc, and you can also take a look at POC patch if you interested
in implementation details. "Background" part could be skipped if you have understandings of
preemption logic implementation.

What I have done in the POC patch:
- Add a "dryrun" flag to CapacityScheduler to do preemption checks.
- Implemented PreemptionMonitor (major code comes from ProportionalCapacityPreemptionPolicy).
See ProportionalCapacityMonitorPolicy.
- Implemented PreemptionManager, it stores relationship between demanding apps (which comes
from under-satisifed queues) and to-be-preempted containers (which comes from over-satisifed
queues).
And also it can make preemption decision according to ideal allocation (calculated by PreemptionMonitor).
- Add a simple unit test to demonstrate how it works from end-to-end. (Allocate a 4G container
which needs to preempt 4 * 1G container on a same node).

TODO items:
- Reserved container preemption should be also considered
- Optimize how to select to-be-preempt containers, selected containers should have minimal
impact
- Similar to above, when PreemptionManager found a better combination of containers to be
preempted (such as containers from lower-priority applications), it should be able to update
preemption target(s).
- Many other corner cases.

Any review comments / suggestions are welcome! And please feel free to let me know if you
have any questions. 
+ [~curino], [~jlowe], [~eepayne], [~sunilg]

Thanks,

> CapacityScheduler: Improve preemption to preempt only those containers that would satisfy
the incoming request
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption is more
effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality (I only
want to use rack-1) / node-constraints (YARN-3409) / black-list (I don't want to use rack1
and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), cross applicaiton
preemption (such as priority-based (YARN-1963) / fairness-based (YARN-3319)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message