hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-4425) Pluggable sharing policy for Partition Node Label resources
Date Thu, 14 Apr 2016 21:39:25 GMT

    [ https://issues.apache.org/jira/browse/YARN-4425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241962#comment-15241962
] 

Wangda Tan commented on YARN-4425:
----------------------------------

[~Naganarasimha],

Thanks, let me rephrase what we shouldn't change and what we could change:

*Following protocol that shouldn't change:*
- Allocated resource to app should on the same partition of what requested by the app.
We definitely should avoid defining protocol to allow sharing between partitions, that's not
intuitive.

If you look at non-exclusive node partition again, it is not resource sharing between partition.
We previously decided that empty node-label-expression means "prefers on default node partition,
but anywhere else is also acceptable". I admit this is not easy to understand as well, but
at least better than "sharing between partition". If I have chance to re-do the non-exclusive
partition, I will not define protocols like that.

I suggest you can take look at YARN-4902, the better API to support non-exclusive node partition
should be:

{code}
Delayed-OR [
- Give me default partition, 
(Wait for x mins)
- Any partition is acceptable. 
]
{code}

Similarly, if you want to support "only partition-B can share partition-A's idle resource"
should be:
{code}
Delayed-OR [
- Give me partition=B, 
(Wait for x mins)
- partition=A is also acceptable 
]
{code}

*What we could change:*
- Adjust queue's shares on partitions based on policy
I don't think this will be harder than new sharing policy proposed by this JIRA. If we want
to do this internally in YARN, a couple of changes/refactorings of CS required to support
dynamically configuration of queue capacities.
- Update nodes' partition externally based on policy:
bq. So practically this needs to be handled externally instead based on policy i try to acheive
the same thing transparently from inside. Thoughts?
You can use centralized-delegation to update partitions inside YARN.

> Pluggable sharing policy for Partition Node Label resources
> -----------------------------------------------------------
>
>                 Key: YARN-4425
>                 URL: https://issues.apache.org/jira/browse/YARN-4425
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler, resourcemanager
>            Reporter: Naganarasimha G R
>            Assignee: Naganarasimha G R
>         Attachments: ResourceSharingPolicyForNodeLabelsPartitions-V1.pdf, ResourceSharingPolicyForNodeLabelsPartitions-V2.pdf,
YARN-4425.20160105-1.patch
>
>
> As part of support for sharing NonExclusive Node Label partitions in YARN-3214, NonExclusive
partitions are shared only to Default Partitions and also have fixed rule when apps in default
partitions makes use of resources of any NonExclusive partitions.
> There are many scenarios where in we require pluggable policy like MutliTenant, Hierarchical
etc.. where in each partition can determine when they want to share the resources to other
paritions and when other partitions wants to use resources from others
> More details in the attached document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message