hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xianyin Xin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4120) FSAppAttempt.getResourceUsage() should not take preemptedResource into account
Date Mon, 14 Sep 2015 02:30:46 GMT

    [ https://issues.apache.org/jira/browse/YARN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14742812#comment-14742812
] 

Xianyin Xin commented on YARN-4120:
-----------------------------------

Hi [~kasha], [~asuresh], [~ashwinshankar77], now both the preemption logic and resource allocation
logic uses {{comparator}} to sort the {{Schedulables}}. I think we have to introduce a different
comparator to separate {{usage}} and {{usage - preemption}}, just as the patch in YARN-4134.
There're also some discussion on changing {{Comparator.compare()}} in YARN-3453. I think for
a collection of comparables, we can use different comparators to compare different attributes
for different purpose. Any thoughts?

> FSAppAttempt.getResourceUsage() should not take preemptedResource into account
> ------------------------------------------------------------------------------
>
>                 Key: YARN-4120
>                 URL: https://issues.apache.org/jira/browse/YARN-4120
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>            Reporter: Xianyin Xin
>
> When compute resource usage for Schedulables, the following code is envolved,
> {{FSAppAttempt.getResourceUsage}},
> {code}
> public Resource getResourceUsage() {
>   return Resources.subtract(getCurrentConsumption(), getPreemptedResources());
> }
> {code}
> and this value is aggregated to FSLeafQueues and FSParentQueues. In my opinion, taking
{{preemptedResource}} into account here is not reasonable, there are two main reasons,
> # it is something in future, i.e., even though these resources are marked as preempted,
it is currently used by app, and these resources will be subtracted from {{currentCosumption}}
once the preemption is finished. it's not reasonable to make arrange for it ahead of time.

> # there's another problem here, consider following case,
> {code}
>             root
>            /    \
>       queue1   queue2
>       /    \
> queue1.3, queue1.4
> {code}
> suppose queue1.3 need resource and it can preempt resources from queue1.4, the preemption
happens in the interior of queue1. But when compute resource usage of queue1, {{queue1.resourceUsage
= it's_current_resource_usage - preemption}} according to the current code, which is unfair
to queue2 when doing resource allocating.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message