hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yufei Gu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6151) FS Preemption doesn't filter out queues which cannot be preempted
Date Mon, 06 Feb 2017 21:08:42 GMT

    [ https://issues.apache.org/jira/browse/YARN-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854730#comment-15854730

Yufei Gu commented on YARN-6151:

Uploaded one simple solution is that before we choose which queue to preempt, we check if
this queue can be preempted. For leaf queue, we just check if the queue uses more resources
than its fair share. For parent queue, we need to recursively go down to each level to check.
There might be a performance concern, but considering the performance is still acceptable
before YARN-5969 which did recursively get resource usage anyway, it should be fine.

> FS Preemption doesn't filter out queues which cannot be preempted
> -----------------------------------------------------------------
>                 Key: YARN-6151
>                 URL: https://issues.apache.org/jira/browse/YARN-6151
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.8.0
>            Reporter: Yufei Gu
>            Assignee: Yufei Gu
>         Attachments: YARN-6151.001.patch
> This is preemption bug happens before 2.8.0, which also described in YARN-3405.
> Queue hierarchy described as below:
> {noformat}
>                   root
>                /         \
>        queue-1          queue-2       
>           /      \
> queue-1-1     queue-1-2
> {noformat}
> Assume cluster resource is 100 and all queues have same weights.
> # queue-1-1 and queue-2 has apps. Each get 50 usage and 50 fairshare. 
> # When queue-1-2 is active, supposedly it will preempt 25 from queue-1-1, but this doesn't
happen because preemption happens top-down, queue-2 could be the preemption candidate as long
as queue-2 is less needy than queue-1, and queue-2 doesn't exceed the fair share which means
preemption won't happen.
> We need to filter out queue-2 since it isn't a valid candidate.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message