hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4425) Pluggable sharing policy for Partition Node Label resources
Date Fri, 08 Apr 2016 00:42:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15231431#comment-15231431

Wangda Tan commented on YARN-4425:

[~Naganarasimha], [~xinxianyin],
Read the doc and took a very high level look at the patch, sorry for the huge delays. Some

The new added policy seems like a backdoor of scheduler's capacity management: when under
"NON_EXCLUSIVE" mode, scheduler completely depends on configured policy to decide who can
get next resources.

And we need to make API of policy with more clear semantic: In existing scheduler, resource
will be allocated on requested partition, and only request.partition = "" can get chance to
allocate on other partitions when non-exclusive criteria is met.
In the new API, resource could be allocated on any partition regardless of requested partition.
(depends on different policy implementation). Which will be conflict to our existing APIs
for node partitions.

To me, sharing resource between partitions itself is not a clear API:
You can say partition A has total resource = 100G, partition B has total resource = 200G.
But you cannot say: "under some conditions, partition A can use idle resources from partition
B" -- because partition is not the entity which will consume resources.

Instead, it will be more clear to me to say:
1) Queue's shares of partitions could be dynamically adjusted, OR
2) Node's partition could be dynamically update

> Pluggable sharing policy for Partition Node Label resources
> -----------------------------------------------------------
>                 Key: YARN-4425
>                 URL: https://issues.apache.org/jira/browse/YARN-4425
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler, resourcemanager
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>         Attachments: ResourceSharingPolicyForNodeLabelsPartitions-V1.pdf, ResourceSharingPolicyForNodeLabelsPartitions-V2.pdf,
> As part of support for sharing NonExclusive Node Label partitions in YARN-3214, NonExclusive
partitions are shared only to Default Partitions and also have fixed rule when apps in default
partitions makes use of resources of any NonExclusive partitions.
> There are many scenarios where in we require pluggable policy like MutliTenant, Hierarchical
etc.. where in each partition can determine when they want to share the resources to other
paritions and when other partitions wants to use resources from others
> More details in the attached document.

This message was sent by Atlassian JIRA

View raw message