hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
Date Wed, 26 Apr 2017 22:19:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985661#comment-15985661

Wangda Tan commented on YARN-5881:

Thanks [~sunilg] for the POC patch, general approach looks good, several structural/cosmetic

1) For the quotas (Like effective min/max, configured min/max, etc.) I think it is better
put all of then to one class. Because it can handle following things better: a. for not existed
label, it can return a zero resource instead of null. b. can have a separate lock.

What I'm thinking now is moving common methods / fields from ResourceUsage to a common class,
and leave all type-specifc methods (such as setCachedUsed) in the subclass.

And we can create a a new class (Such as QueueResourceQuotas), which extend the base class,
but with some methods to read/write queue resource quota types.

Downside of the approach is, resources associated to one partition is determined by ResourceUsage#ResourceType.
Without lots of changes, for each partition, it will include many unnecessary resource types
(for example used_resource is unnecessary for ResourceQuotas). It looks not a big problem
to me since accessing items in an array is very fast and number of partitions is limited.

2) Can we move {{loadResourceConstraintByLabelsFromConf}} from {{CSQueueUtils}} to AbstractCSQueue?
Since it is more related to the queue and its parent queue, with this change lots of input
parameters of {{loadResourceConstraintByLabelsFromConf}} can be removed.

3) For set effective min/max Resource, do you think we can only do it inside {{LeafQueue/ParentQueue#updateClusterResource}}?
Now {{computeEffectiveResourcesForAllQueues}} creates a separate recursive call, I think it
will be better to put all queue-resource-related changes inside {{LeafQueue/ParentQueue#updateClusterResource}}.

Please let me know your thoughts. And for the next patch, it's better if you can update all
scheduler paths to use getEffective*.

> Enable configuration of queue capacity in terms of absolute resources
> ---------------------------------------------------------------------
>                 Key: YARN-5881
>                 URL: https://issues.apache.org/jira/browse/YARN-5881
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Sean Po
>            Assignee: Wangda Tan
>         Attachments: YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf,
YARN-5881.v0.patch, YARN-5881.v1.patch
> Currently, Yarn RM supports the configuration of queue capacity in terms of a proportion
to cluster capacity. In the context of Yarn being used as a public cloud service, it makes
more sense if queues can be configured absolutely. This will allow administrators to set usage
limits more concretely and simplify customer expectations for cluster allocation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message