hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
Date Thu, 14 Jan 2016 18:04:39 GMT

    [ https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15098526#comment-15098526
] 

Sunil G commented on YARN-4108:
-------------------------------

Thank you [~leftnoteasy] for the detailed updated design doc. Few comments or queries

1. In current PCPP, we first preempt all reserved continers from all applications in a queue
if its overallocated. After introduding "killable containers", so we will have some containers
which are reserved or running. Am I correct? or reserved containers are planning to handled
differently?

Possible cases:	
	- queueA's appA has made a reservation on node1. now queueB has demand, and we can unreserve
appA's container requests from node1 and queueB's app can allocate some containers there.
	- or queueB's app can now reserve a container there. This case I think is explained by the
design doc.
Basically we can still try to preempt reserved container first.

2. If I understood correctly, "killable containers" will be triggered with preeempt event
only if a proper allocation can happen for target application (from underserving queue). 
	- So do we send preempt_container event to AM here at  this point? This can make a delay
of 15secs, so if by some chance scheduling footprint is changed (some other NMs freed space),
we may some overkill. (I guess this can happen now also).
	- 15secs later, RM do forcekill to these containers. Is there any change for this approach
in this new design?

because as per doc, its mentioned that we need to preempt only if we can reserve. So I am
slightly confused here.

3. To cancel "killable container", i think PCPP will take the call by waiting for some interval.
So some new configuration is needed for this?

4. I would like to have some freedom in selecting conatiners (marking) for preemption. A simple
sorting based on submission time or priority seems limited approach. Could we have some interface
here so that we can plugin user specific comparision cases.
	- submission time
	- priority
	- demand based etc may be

> CapacityScheduler: Improve preemption to preempt only those containers that would satisfy
the incoming request
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-4108-design-doc-V3.pdf, YARN-4108-design-doc-v1.pdf, YARN-4108-design-doc-v2.pdf,
YARN-4108.poc.1.patch, YARN-4108.poc.2-WIP.patch
>
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption is more
effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality (I only
want to use rack-1) / node-constraints (YARN-3409) / black-list (I don't want to use rack1
and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), cross applicaiton
preemption (such as priority-based (YARN-1963) / fairness-based (YARN-3319)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message