hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karthik Kambatla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-2394) Fair Scheduler : ability to configure fairSharePreemptionThreshold per queue
Date Thu, 28 Aug 2014 15:59:08 GMT

    [ https://issues.apache.org/jira/browse/YARN-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113892#comment-14113892

Karthik Kambatla commented on YARN-2394:

If the user queues inherit from the parent and have a threshold of 1, the scheduler would
never preempt unless the queue has over its fairshare. If we are preempting resources that
are over the fairshare, I think it is fair game. We can improve choosing the victim, but we
can work on it in another JIRA.

OTOH, choosing default-threshold of 0.5 for user queues can lead to:
# one of the user queues starve (stay under fairshare) while other user queues are higher
than threshold
# depending on preemption logic (being revisited in YARN-2154), it might take a while to reach
the threshold for parent queues

> Fair Scheduler : ability to configure fairSharePreemptionThreshold per queue
> ----------------------------------------------------------------------------
>                 Key: YARN-2394
>                 URL: https://issues.apache.org/jira/browse/YARN-2394
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: fairscheduler
>            Reporter: Ashwin Shankar
>            Assignee: Wei Yan
>         Attachments: YARN-2394-1.patch, YARN-2394-2.patch
> Preemption based on fair share starvation happens when usage of a queue is less than
50% of its fair share. This 50% is hardcoded. We'd like to make this configurable on a per
queue basis, so that we can choose the threshold at which we want to preempt. Calling this
config "fairSharePreemptionThreshold". 

This message was sent by Atlassian JIRA

View raw message