hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ashwin Shankar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2394) Fair Scheduler : ability to configure fairSharePreemptionThreshold per queue
Date Thu, 28 Aug 2014 04:29:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113323#comment-14113323

Ashwin Shankar commented on YARN-2394:

[~ywskycn], [~kasha]
Inheriting fairSharePreemptionThreshold wouldn't work for us. Do you guys have a use case
for inheriting this config ?
We have the following requirement :
root.parentUserQueue : fair share = 80% fairSharePreemptionThreshold=1
root.parentUserQueue.user(1-100) : defaultFairSharePreemptionThreshold=0.5

Basically when parentUserQueue is starved, we want to preempt more aggressively(threshold=1)
from its siblings under root.
While when a user queue is starved, we want to preempt less aggressively from other user queues(assuming
other user queues are above fair share).
We can do this by setting fairSharePreemptionThreshold=1 at parentUserQueue and defaultFairSharePreemptionThreshold=0.5
which would be used by user queues.
You might say that we can override fairSharePreemptionThreshold to 0.5 in the user queues,
but these queues are created on the fly when a new user 
submits an app and NOT through the alloc xml and there are hundreds of them, so its not very
feasible to override each one of them in the config.

Would it be okay if we stick to version-1 of the patch which doesn't inherit threshold ? 

> Fair Scheduler : ability to configure fairSharePreemptionThreshold per queue
> ----------------------------------------------------------------------------
>                 Key: YARN-2394
>                 URL: https://issues.apache.org/jira/browse/YARN-2394
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: fairscheduler
>            Reporter: Ashwin Shankar
>            Assignee: Wei Yan
>         Attachments: YARN-2394-1.patch, YARN-2394-2.patch
> Preemption based on fair share starvation happens when usage of a queue is less than
50% of its fair share. This 50% is hardcoded. We'd like to make this configurable on a per
queue basis, so that we can choose the threshold at which we want to preempt. Calling this
config "fairSharePreemptionThreshold". 

This message was sent by Atlassian JIRA

View raw message