Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 26 Apr 2017 22:19:04 +0000 (UTC)
From: "Wangda Tan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13020654.1479168100000.60444.1493245144211@Atlassian.JIRA>
In-Reply-To: <JIRA.13020654.1479168100000@Atlassian.JIRA>
References: <JIRA.13020654.1479168100000@Atlassian.JIRA> <JIRA.13020654.1479168100687@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-5881) Enable configuration of queue
 capacity in terms of absolute resources
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 26 Apr 2017 22:19:09 -0000


    [ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985661#comment-15985661 ] 

Wangda Tan commented on YARN-5881:
----------------------------------

Thanks [~sunilg] for the POC patch, general approach looks good, several structural/cosmetic suggestions: 

1) For the quotas (Like effective min/max, configured min/max, etc.) I think it is better put all of then to one class. Because it can handle following things better: a. for not existed label, it can return a zero resource instead of null. b. can have a separate lock.

What I'm thinking now is moving common methods / fields from ResourceUsage to a common class, and leave all type-specifc methods (such as setCachedUsed) in the subclass.

And we can create a a new class (Such as QueueResourceQuotas), which extend the base class, but with some methods to read/write queue resource quota types.

Downside of the approach is, resources associated to one partition is determined by ResourceUsage#ResourceType. Without lots of changes, for each partition, it will include many unnecessary resource types (for example used_resource is unnecessary for ResourceQuotas). It looks not a big problem to me since accessing items in an array is very fast and number of partitions is limited.

2) Can we move {{loadResourceConstraintByLabelsFromConf}} from {{CSQueueUtils}} to AbstractCSQueue? Since it is more related to the queue and its parent queue, with this change lots of input parameters of {{loadResourceConstraintByLabelsFromConf}} can be removed.

3) For set effective min/max Resource, do you think we can only do it inside {{LeafQueue/ParentQueue#updateClusterResource}}? Now {{computeEffectiveResourcesForAllQueues}} creates a separate recursive call, I think it will be better to put all queue-resource-related changes inside {{LeafQueue/ParentQueue#updateClusterResource}}. 

Please let me know your thoughts. And for the next patch, it's better if you can update all scheduler paths to use getEffective*.

> Enable configuration of queue capacity in terms of absolute resources
> ---------------------------------------------------------------------
>
>                 Key: YARN-5881
>                 URL: https://issues.apache.org/jira/browse/YARN-5881
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Sean Po
>            Assignee: Wangda Tan
>         Attachments: YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf, YARN-5881.v0.patch, YARN-5881.v1.patch
>
>
> Currently, Yarn RM supports the configuration of queue capacity in terms of a proportion to cluster capacity. In the context of Yarn being used as a public cloud service, it makes more sense if queues can be configured absolutely. This will allow administrators to set usage limits more concretely and simplify customer expectations for cluster allocation.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org