hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4333) Fair scheduler should support preemption within queue
Date Thu, 18 Feb 2016 02:14:18 GMT

    [ https://issues.apache.org/jira/browse/YARN-4333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15151595#comment-15151595

Karthik Kambatla commented on YARN-4333:

Thanks a lot for filing this JIRA and working on it. The preemption logic is broken at different
places. We have other JIRAs where preemption between sibling queues doesn't work well. 

Quickly skimmed through the patch here. Few high-level comments on the patch and preemption
in general:
# Today, all preemption is top-down. When we encounter a {{Schedulable}} (app/queue) that
is starved, we should trigger preemption from the closest parent that is not starved. Preempting
resources from another application under the same leaf-queue that is over its fairshare is
automatically taken care of.
# The preemptionTimeout and preemptionThreshold are defined for queues and not apps. We should
decide on what values we want to use for apps. May be, our best bet is to use the ones the
corresponding leaf-queue uses? 

[~ashwinshankar77], [~sandyr] - would like to hear your thoughts as well. 

> Fair scheduler should support preemption within queue
> -----------------------------------------------------
>                 Key: YARN-4333
>                 URL: https://issues.apache.org/jira/browse/YARN-4333
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>    Affects Versions: 2.6.0
>            Reporter: Tao Jie
>            Assignee: Tao Jie
>         Attachments: YARN-4333.001.patch, YARN-4333.002.patch, YARN-4333.003.patch
> Now each app in fair scheduler is allocated its fairshare, however  fairshare resource
is not ensured even if fairSharePreemption is enabled.
> Consider: 
> 1, When the cluster is idle, we submit app1 to queueA,which takes maxResource of queueA.
> 2, Then the cluster becomes busy, but app1 does not release any resource, queueA resource
usage is over its fairshare
> 3, Then we submit app2(maybe with higher priority) to queueA. Now app2 has its own fairshare,
but could not obtain any resource, since queueA is still over its fairshare and resource will
not assign to queueA anymore. Also, preemption is not triggered in this case.
> So we should allow preemption within queue, when app is starved for fairshare.

This message was sent by Atlassian JIRA

View raw message