hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6081) LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container
Date Wed, 11 Jan 2017 11:09:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818018#comment-15818018
] 

Sunil G commented on YARN-6081:
-------------------------------

Thanks [~leftnoteasy]. Good catch!

We decrement pending resources only if container is allocated (not reserved). So ideally we
have to deduct reserved memory from pending resource if any. Ideally makes sense for me.

Few comments:
1. {{getTotalPendingResourcesConsideringUserLimit}}, Not part of this patch. Could have a
java doc comment there as well. So it ll be make javadoc also more  better?
2. 
{code}
	        Resource pending = app.getAppAttemptResourceUsage().getPending(
	            partition);
	        if (deductReservedFromPending) {
	          pending = Resources.subtract(pending,
	              app.getAppAttemptResourceUsage().getReserved(partition));
	        }
{code}

I have one doubt here. {{pending}} holds a reference of pending resource of appAttemptResource
usage. Inside {{if(deductReservedFromPending)}} block, that reference is getting updated.
Is that intentional?

3.  
{code}
	        pending = Resources.max(resourceCalculator, lastClusterResource,
	            pending, Resources.none());
{code}
A quick doubt. Why are we using lastClusterResource here?

4. {{testPreemptionNotHappenForSingleReservedQueue}}, comment near verify block is confusing.
5. In {{testPendingResourcesConsideringUserLimit}}, could we also try to assert the app's
pending and reserved too?


> LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending
to avoid unnecessary preemption of reserved container
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6081
>                 URL: https://issues.apache.org/jira/browse/YARN-6081
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-6081.001.patch
>
>
> While doing YARN-5864 tests, found an issue when a queue's reserved > pending. PreemptionResourceCalculator
will preempt reserved container even if there's only one active queue in the cluster. 
> To fix the problem, we need to deduct reserved from pending when getting total-pending
resource for LeafQueue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message