hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7237) Cleanup usages of ResourceProfiles
Date Tue, 26 Sep 2017 19:03:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181325#comment-16181325

Wangda Tan commented on YARN-7237:


I think the default for minimum / maximum is a reasonable config: for a new resource type,
unless if it's mandatory, minimum should be 0. And the effective maximum resource value will
be capped by maximum resource value on each node. Admin can set some limits when necessary.

As our discussion with [~templedf] offline, we may need another cleanup to remove profile
from ResourceRequest, I felt this patch should be still valid since it includes some basic
cleanups. We can work on the next patch on top of this one, and I want to get this done sooner
than later.

> Cleanup usages of ResourceProfiles
> ----------------------------------
>                 Key: YARN-7237
>                 URL: https://issues.apache.org/jira/browse/YARN-7237
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>            Priority: Critical
>         Attachments: YARN-7237.001.patch, YARN-7237.002.patch, YARN-7237.003.patch
> While doing tests, there're a couple of issues:
> 1) When use {{ProfileCapability#getProfileCapabilityOverride}}, it does overwrite of
whatever specified in resource-profiles.json when value >= 0. Which is different from javadocs
of {{ProfileCapability}} 
> bq. For example, if you have a resource profile "small" that maps to <4096M, 2 cores,
1 gpu> and you set the capability override to <8192M, 0 cores, 0 gpu>, then the actual
resource allocation on the ResourceManager will be <8192M, 2 cores, 1 gpu>
> To me, the correct behavior should do overwrite when value > 0. The reason is, by
default resource value will be set to 0, For example, assume we have a profile {{"a" = (mem=3,
vcore=5, res_1=7)}}, and create a capability-overwrite (capability = new resource(8). The
final result should be (mem=8, vcore=5, res_1=7), instead of (mem=8, vcore=0, res_1=0).
> 2) ResourceProfileManager now loads minimum/maximum profile from config file (resource-profiles.json),
to me this is not correct because minimum/maximum allocation for each resource types are already
specified inside {{resource-types.xml}}. We should always use {{ResourceUtils#getResourceTypesMinimum/MaximumAllocation}}
to get from resource-types.xml and yarn-site.xml. This value will be added to profiles so
client can get these configs.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message