hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
Date Wed, 26 Apr 2017 22:19:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985661#comment-15985661
] 

Wangda Tan commented on YARN-5881:
----------------------------------

Thanks [~sunilg] for the POC patch, general approach looks good, several structural/cosmetic
suggestions: 

1) For the quotas (Like effective min/max, configured min/max, etc.) I think it is better
put all of then to one class. Because it can handle following things better: a. for not existed
label, it can return a zero resource instead of null. b. can have a separate lock.

What I'm thinking now is moving common methods / fields from ResourceUsage to a common class,
and leave all type-specifc methods (such as setCachedUsed) in the subclass.

And we can create a a new class (Such as QueueResourceQuotas), which extend the base class,
but with some methods to read/write queue resource quota types.

Downside of the approach is, resources associated to one partition is determined by ResourceUsage#ResourceType.
Without lots of changes, for each partition, it will include many unnecessary resource types
(for example used_resource is unnecessary for ResourceQuotas). It looks not a big problem
to me since accessing items in an array is very fast and number of partitions is limited.

2) Can we move {{loadResourceConstraintByLabelsFromConf}} from {{CSQueueUtils}} to AbstractCSQueue?
Since it is more related to the queue and its parent queue, with this change lots of input
parameters of {{loadResourceConstraintByLabelsFromConf}} can be removed.

3) For set effective min/max Resource, do you think we can only do it inside {{LeafQueue/ParentQueue#updateClusterResource}}?
Now {{computeEffectiveResourcesForAllQueues}} creates a separate recursive call, I think it
will be better to put all queue-resource-related changes inside {{LeafQueue/ParentQueue#updateClusterResource}}.


Please let me know your thoughts. And for the next patch, it's better if you can update all
scheduler paths to use getEffective*.

> Enable configuration of queue capacity in terms of absolute resources
> ---------------------------------------------------------------------
>
>                 Key: YARN-5881
>                 URL: https://issues.apache.org/jira/browse/YARN-5881
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Sean Po
>            Assignee: Wangda Tan
>         Attachments: YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf,
YARN-5881.v0.patch, YARN-5881.v1.patch
>
>
> Currently, Yarn RM supports the configuration of queue capacity in terms of a proportion
to cluster capacity. In the context of Yarn being used as a public cloud service, it makes
more sense if queues can be configured absolutely. This will allow administrators to set usage
limits more concretely and simplify customer expectations for cluster allocation.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message