hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5293) Disable preemption per queue per partition
Date Thu, 30 Jun 2016 09:48:10 GMT

    [ https://issues.apache.org/jira/browse/YARN-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15356832#comment-15356832

Sunil G commented on YARN-5293:

Thanks [~jhung] for a detailed idea about the configurations and its priorities.
I am thinking about the advantages of disabling preemption for per-label-per-queue level.
Since labels can give a physical categorization of nodes based on certain types (upto the
admin to decide), we can think that one specific queue capacity can be splitted/viewed as
few certain types of nods. So my *highGPU* label nodes for the queue can be disabled for preemption
when compared with my *lowGPU*, thus keeping only some low priority resources for preemption.
This can be viewed from both angle. I am saving some of my critical resources from getting
preempted but still gives a certain percentage of resource which are of low priority to preempt.
But there is a fair level balance needed, and it may be tough to maintain. Few points
- *labelA* for queueA is now disabled for preemption and consider this queue is over-utilizing
resources from *labelA*. *labelA* is configured in queueB also which is under utilized. So
ideally *queueA-labelA* has to be preempted to satisfy queueB's need. This will be blocked
- If only one label is configured per-queue, its same queue level disabling
- With label exclusivity disabled, same label can be used for NO_LABEL cases too for a queue.
So if this label is disabled from preemption, indirectly NO_LABEL resources also will be saved.
Such cases are hard to explain or propagate to user/admin level for clarity.

I think more discussions are good here. cc/[~leftnoteasy], [~Naganarasimha Garla], [~rohithsharma]

> Disable preemption per queue per partition
> ------------------------------------------
>                 Key: YARN-5293
>                 URL: https://issues.apache.org/jira/browse/YARN-5293
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>            Reporter: Jonathan Hung
> [YARN-2056|https://issues.apache.org/jira/browse/YARN-2056] added support for disabling
preemption at queue level. [YARN-2498|https://issues.apache.org/jira/browse/YARN-2498] added
support for per partition preemption. It would be useful to be able to disable preemption
per partition per queue, so that if a queue Q has access to two labels A and B, we can disable
preemption for Q in A but not in B.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message