hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Payne (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
Date Wed, 07 Sep 2016 19:13:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15471536#comment-15471536
] 

Eric Payne commented on YARN-4945:
----------------------------------

[~leftnoteasy] and [~sunilg], I'm still concerned about having the intra-queue preemption
policies adding containers to the {{selectedCandidates}} list if the inter-queue policies
have already added containers. In that case, the containers selected by the intra-queue policies
may not go back to the correct queue. Consider this use case:

Queues (all are preemptable):
||Queue Name||Guaranteed Resources||Max Resources||{{total_preemption_per_round}}||
|root|200|200|0.1|
|QUEUE1|100|200|0.1|
|QUEUE2|100|200|0.1|

# {{User1}} starts {{App1}} on {{QUEUE1}} and uses all 200 resources. These containers are
long-running and will not be released any time soon:
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending Resources||Selected
For Preemption||
|QUEUE1|User1|App1|1|200|0|0|
# {{User2}} starts {{App2}} on {{QUEUE2}} and requests 100 resources:
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending Resources||Selected
For Preemption||
|QUEUE1|User1|App1|1|200|0|0|
|QUEUE2|User2|App2|1|0|100|0|
# {{User1}} starts {{App3}} at a high priority on {{QUEUE1}} and requests 50 resources:
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending Resources||Selected
For Preemption||
|QUEUE1|User1|App1|1|200|0|0|
|QUEUE1|User1|App3|10|0|50|0|
|QUEUE2|User2|App2|1|0|100|0|
# Since {{total_preemption_per_round}} is 0.1, only 10% of the needed resources will be selected
per round. So, the inter-queue preemption policies select 10 resources to be preempted from
{{App1}}.
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending Resources||Selected
For Preemption||
|QUEUE1|User1|App1|1|200|0|10|
|QUEUE1|User1|App3|10|0|50|0|
|QUEUE2|User2|App2|1|0|100|0|
# Then, the priority-intra-queue preemption policy selects 5 more resources to be preempted
from {{App1}}.
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending Resources||Selected
For Preemption||
|QUEUE1|User1|App1|1|200|0|15|
|QUEUE1|User1|App3|10|0|50|0|
|QUEUE2|User2|App2|1|0|100|0|
# At this point, 15 resources are preempted from {{App1}}.
# Since {{QUEUE2}} is asking for 100 resources, and is extremely underserved (from an inter-queue
point of view), the capacity scheduler gives all 15 resources to {{QUEUE2}}, and the priority
inversion remains in {{QUEUE1}}.
||Queue Name||User Name||App Name||App Priority||Used Resources||Pending Resources||Selected
For Preemption||
|QUEUE1|User1|App1|1|185|15|0|
|QUEUE1|User1|App3|10|0|50|0|
|QUEUE2|User2|App2|1|15|85|0|

This is why I am concerned that when containers are already selected by the inter-queue preemption
policies, it may not be beneficial to have the intra-queue policies preempt containers as
well.


> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
>                 Key: YARN-4945
>                 URL: https://issues.apache.org/jira/browse/YARN-4945
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>         Attachments: Intra-Queue Preemption Use Cases.pdf, IntraQueuepreemption-CapacityScheduler
(Design).pdf, YARN-2009-wip.2.patch, YARN-2009-wip.patch, YARN-2009-wip.v3.patch, YARN-2009.v0.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to support features
like:
> YARN-2009. YARN-2113. YARN-4781.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message