hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4945) [Umbrella] Capacity Scheduler Preemption Within a queue
Date Wed, 31 Aug 2016 23:55:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-4945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453773#comment-15453773
] 

Wangda Tan commented on YARN-4945:
----------------------------------

Hi Sunil,

Thanks for update, I haven't dig into too many details of this patch yet, I majorly looked
at interactions between preemptable-resource-calculator and candidate-selector.

Some comments so far:

1) What is intraQueuePreemptionCost? And what is AppPriorityComparator?

2) IntraQueuePreemptableResourceCalculator:
2.1) For the queue-level ideal allocation, is it enough to trust result from previous policies?
PreemptableResourceCalculator saves per-queue ideal allocation to PCPP#queueToPartitions.
I think we don't need add extra logic in the IntraQueuePreemptableResourceCalculator to resursively
calculate it. Correct?

2.2 computeIntraQueuePreemptionDemand:
- {{if (tq.intraQueuePreemptionCalculationDone == true)}}, is this always false?
- Should we lock the queue inside {{for (LeafQueue leafQUeue : queues)}}?
- return value of getResourceDemandFromAppsPerQueue is not consumed by anyone.
- TempAppPerQueue is not used by anyone

Since the logic looks incompleted yet, here's some thoughts about implementation/overall-code-structure
from my side, hope it may help you.

{code}
IntraQueuePreemptableResourceCalculator {
	1. Uses ideal resource calculated by previous policies. 
	2. Compute per-app ideal/preemptable resource according to per-queue policies. (Stored in
TempAppPerPartition)
}

IntraQueueCandidateSelector {
	1. Invoke IntraQueueCandidateSelector to calculate ideal/preemptable of apps
	2. Use CapacitySchedulerPreemptionUtils.deductPreemptableResourcesBasedSelectedCandidates
to deduct preemptable resource for already selected containers.

	for (leafqueue from most underserved) {
		for (apps in reserve order) {
			if (app.preemptable > 0) {
				// Select container logic.
			}
		}
	}
}
{code}

> [Umbrella] Capacity Scheduler Preemption Within a queue
> -------------------------------------------------------
>
>                 Key: YARN-4945
>                 URL: https://issues.apache.org/jira/browse/YARN-4945
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>         Attachments: Intra-Queue Preemption Use Cases.pdf, IntraQueuepreemption-CapacityScheduler
(Design).pdf, YARN-2009-wip.2.patch, YARN-2009-wip.patch
>
>
> This is umbrella ticket to track efforts of preemption within a queue to support features
like:
> YARN-2009. YARN-2113. YARN-4781.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message