hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
Date Thu, 01 Sep 2016 10:03:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454965#comment-15454965
] 

Sunil G commented on YARN-4945:
-------------------------------

Thanks [~leftnoteasy]
bq.What is intraQueuePreemptionCost? And what is AppPriorityComparator?
Yes, there are not used. I will do a clean up in next patch to remove unnecessary code snippets.

bq.For the queue-level ideal allocation is it enough to trust result from previous policies?
If we are planning for normalization (I want to introduce in next patch after doing some more
tests with SLS and real time), then we need to add up or subtract resources which are preempted
from previous run. And this is needed to done per queue. Hence I kept the code. I can look
into this and see how much more i can reuse as possible.

bq.if (tq.intraQueuePreemptionCalculationDone == true), is this always false?
intraQueuePreemptionCalculationDone will be set as false during init. We iterate over partitions,
and there are chances that we may have multiple labels in a queue. Since I am iterating over
partitions and then look into queues where this partition has some capacity, we may have same
queue across partition. I was calculating resource demand like partition -> queue level.
To avoid duplicate calculation, i use this variable. 

bq.Should we lock the queue inside for (LeafQueue leafQUeue : queues)?
Make sense for me. I will handle this case.

bq.return value of getResourceDemandFromAppsPerQueue is not consumed by anyone.
I have cleaned up the code.

bq.TempAppPerQueue is not used by anyone
 I am moving some common logic computation and some storage to here. I will have this in next
patch.

Thanks for sharing the flow. I also planned to do similar approach what you suggested, but
I am doing some tasks together. I feel I can rearrange the code to fit to this code flow.
However I feel deductPreemptableResourcesBasedSelectedCandidates could be done while we loop
over each containers of an app also (we can deduct from resourceToObtain, but do not add same
container again. I handled this scenario already). 


> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
>                 Key: YARN-4945
>                 URL: https://issues.apache.org/jira/browse/YARN-4945
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>         Attachments: Intra-Queue Preemption Use Cases.pdf, IntraQueuepreemption-CapacityScheduler
(Design).pdf, YARN-2009-wip.2.patch, YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to support features
like:
> YARN-2009. YARN-2113. YARN-4781.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message