hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4661) Per-queue preemption policy in FairScheduler
Date Wed, 09 Mar 2016 05:13:40 GMT

    [ https://issues.apache.org/jira/browse/YARN-4661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186519#comment-15186519

Karthik Kambatla commented on YARN-4661:

Interesting thought. In the context of YARN-4752, I am attempting comprehensive changes to
preemption. As discussed in YARN-2154, we think it might best to just preempt any container
if the corresponding application is over its fairshare. The primary goal here is to ensure
the preempted container(s) are all on one node and the resultant resources are actually enough
to allocate the container we are preempting these for. 

In that context, I believe it would quite expensive to also consider a preemption policy.

Do you agree? If yes, may be, we should close this as "Won't fix"? 

> Per-queue preemption policy in FairScheduler
> --------------------------------------------
>                 Key: YARN-4661
>                 URL: https://issues.apache.org/jira/browse/YARN-4661
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler
>    Affects Versions: 2.6.0
>            Reporter: He Tianyi
>            Priority: Minor
> When {{FairScheduler}} needs to preempt container, it tries to find a container by hierachically
sorting and selecting {{AppSchedulable}} with most 'over fairshare' (in {{FairSharePolicy}}),
and pick its latest launched container.
> In some case, strategy above become non-optimal, one may want to kill latest container
(not {{AppSchedulable}}) launched in the queue for better trade-off between fairness and efficiency.
Since most app with over fairshare tend to be started longer ago than other apps, perhaps
even its latent launch container is running quite some time.
> Maybe besides {{policy}}, we make it possible to also specify a {{preemptionPolicy}}
only for selecting container to preempt, without changing scheduling policy.
> For example:
> {quote}
> <queue name="queueA">
>   <preemptionPolicy>fifo</preemptionPolicy>
>   <policy>fair</policy>
> </queue>
> {quote}
> Any suggestions or comments?

This message was sent by Atlassian JIRA

View raw message