hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
Date Thu, 08 Sep 2016 00:34:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15472262#comment-15472262

Wangda Tan commented on YARN-4945:


bq. I actually would like to have the priority policy and the minumum-user-limit-percent policy
be turned on separately. I'm not sure of the best way to do that, but our users don't use
application priority very much.

Now application priority is not very popular because it's not included by any release yet,
but I bet it will be a popular feature in the future. :)

I would prefer to have a unified preemption policy to handle all intra-queue preemption, because:
- We have different combination of intra preemption criteria, like priority, fairness, user-limit,
fifo. We cannot have a separate policy for each different combination. For example, fairness
+ user-limit and priority + user-limit.
- Lots of common part of intra-queue preemption policies, especially after ideal-allocation
resource is calculated for each apps, we have the common logic to select containers.
- We may need different implementation for ideal-allocation resource calculator, one for fairness
and one for fifo, both consider user-limit and priority.

bq. Perhaps CapacitySchedulerConfiguration could have something like:
As I mentioned above, we should enable intra-queue preemption by default for all limits. If
we really need some parameters for better control, we can add them in the future. Otherwise
it cause troubles, for example: consider priority without consider user-limit, excessive preemption
could happen.

> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>                 Key: YARN-4945
>                 URL: https://issues.apache.org/jira/browse/YARN-4945
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>         Attachments: Intra-Queue Preemption Use Cases.pdf, IntraQueuepreemption-CapacityScheduler
(Design).pdf, YARN-2009-wip.2.patch, YARN-2009-wip.patch, YARN-2009-wip.v3.patch, YARN-2009.v0.patch
> This is umbrella ticket to track efforts of preemption within a queue to support features
> YARN-2009. YARN-2113. YARN-4781.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message