Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Wed, 8 Apr 2015 20:30:13 +0000 (UTC)
From: "Wangda Tan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12784835.1427122951000.31178.1428525013132@Atlassian.JIRA>
In-Reply-To: <JIRA.12784835.1427122951000@Atlassian.JIRA>
References: <JIRA.12784835.1427122951000@Atlassian.JIRA>
 <JIRA.12784835.1427122951884@arcas>
Subject: [jira] [Commented] (YARN-3388) Allocation in LeafQueue could get
 stuck because DRF calculator isn't well supported when computing user-limit
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485952#comment-14485952 ] 

Wangda Tan commented on YARN-3388:
----------------------------------

[~nroberts],
Thanks for updating, I took a look at your patch, approach LGTM, but I think node label should be considered in the same JIRA, when doing allocation under a labeled node, user-limit checking in the patch is incorrect. Actually user-limit for exclusive node label is already supported in latest trunk, and user-limit for non-exclusive node label is contains in YARN-3361, I think after YARN-3361, user-limit for node label will be in a good shape.
Would you mind to take a look at computeUserLimit method of the patch attached in YARN-3361?

To support computing consumed-per-partition, User.updateUsageRatio need receive partition as parameter.

Some other comments:
- consumedRatio -> totalDominateConsumed or some other name, it's better to make sum of "dominate" consumed in the name
- consumed -> totalDominatedConsumedByPartition.
- It's better to add a test case to make sure allocation locked descripted in https://issues.apache.org/jira/browse/YARN-3388?focusedCommentId=14376060&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14376060 will not happened.

> Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-3388
>                 URL: https://issues.apache.org/jira/browse/YARN-3388
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacityscheduler
>    Affects Versions: 2.6.0
>            Reporter: Nathan Roberts
>            Assignee: Nathan Roberts
>         Attachments: YARN-3388-v0.patch, YARN-3388-v1.patch
>
>
> When there are multiple active users in a queue, it should be possible for those users to make use of capacity up-to max_capacity (or close). The resources should be fairly distributed among the active users in the queue. This works pretty well when there is a single resource being scheduled.   However, when there are multiple resources the situation gets more complex and the current algorithm tends to get stuck at Capacity. 
> Example illustrated in subsequent comment.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)