Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 11 Jan 2017 11:09:58 +0000 (UTC)
From: "Sunil G (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13033585.1484103975000.708015.1484132998469@Atlassian.JIRA>
In-Reply-To: <JIRA.13033585.1484103975000@Atlassian.JIRA>
References: <JIRA.13033585.1484103975000@Atlassian.JIRA> <JIRA.13033585.1484103975556@arcas>
Subject: [jira] [Commented] (YARN-6081)
 LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct
 reserved from pending to avoid unnecessary preemption of reserved container
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 11 Jan 2017 11:10:01 -0000


    [ https://issues.apache.org/jira/browse/YARN-6081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15818018#comment-15818018 ] 

Sunil G commented on YARN-6081:
-------------------------------

Thanks [~leftnoteasy]. Good catch!

We decrement pending resources only if container is allocated (not reserved). So ideally we have to deduct reserved memory from pending resource if any. Ideally makes sense for me.

Few comments:
1. {{getTotalPendingResourcesConsideringUserLimit}}, Not part of this patch. Could have a java doc comment there as well. So it ll be make javadoc also more  better?
2. 
{code}
	        Resource pending = app.getAppAttemptResourceUsage().getPending(
	            partition);
	        if (deductReservedFromPending) {
	          pending = Resources.subtract(pending,
	              app.getAppAttemptResourceUsage().getReserved(partition));
	        }
{code}

I have one doubt here. {{pending}} holds a reference of pending resource of appAttemptResource usage. Inside {{if(deductReservedFromPending)}} block, that reference is getting updated. Is that intentional?

3.  
{code}
	        pending = Resources.max(resourceCalculator, lastClusterResource,
	            pending, Resources.none());
{code}
A quick doubt. Why are we using lastClusterResource here?

4. {{testPreemptionNotHappenForSingleReservedQueue}}, comment near verify block is confusing.
5. In {{testPendingResourcesConsideringUserLimit}}, could we also try to assert the app's pending and reserved too?


> LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6081
>                 URL: https://issues.apache.org/jira/browse/YARN-6081
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-6081.001.patch
>
>
> While doing YARN-5864 tests, found an issue when a queue's reserved > pending. PreemptionResourceCalculator will preempt reserved container even if there's only one active queue in the cluster. 
> To fix the problem, we need to deduct reserved from pending when getting total-pending resource for LeafQueue.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org