hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Vasudev (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5587) Add support for resource profiles
Date Tue, 25 Oct 2016 14:48:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15605509#comment-15605509

Varun Vasudev commented on YARN-5587:

bq. In Resources, you moved the Suppress deprecation warning from the setMemorySize(long)
method to the setMemory(int). Was that intentional ?

Yes - setMemory is deprecated not setMemorySize. Findbugs complains if I don't supress the

    AMRMClient::ContainerRequest : Wondering if we need to allow a Container request to specify
both a profile name and a Resource (capability). If they do specify both, what does that mean
    Similarly, in the RemoteRequestTable, the RR should be keyed using the Resource (capability)
derived from the profileName.

Good question - my opinion is that profile + capability means take the profile and override
it with the capability. So for example if I say I want profile 'large' with capability <4096M,
2 cores> - that means take all the values from profile large but use 4096M for memory and
2 cores for cpu. This leads to a follow up question - what if you ask for profile 'large'
with capability <4096M, 0 cores>(this will happen because the Resource object doesn't
have 'null' values for memory and vcores)? In this case my proposal is to only use non-zero

The reason I went with this approach is that most users are used to running spark and MR jobs
with memory and cores specified. They'll continue to run these applications the same way.
This approach allows an easy way for admins to turn on resource profiles without affecting
users. However I also accept that the 'only consider non-zero' values for overrides might
seem hackish - I'm absolutely open to alternatives.

> Add support for resource profiles
> ---------------------------------
>                 Key: YARN-5587
>                 URL: https://issues.apache.org/jira/browse/YARN-5587
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: YARN-5587-YARN-3926.001.patch, YARN-5587-YARN-3926.002.patch, YARN-5587-YARN-3926.003.patch,
YARN-5587-YARN-3926.004.patch, YARN-5587-YARN-3926.005.patch
> Add support for resource profiles on the RM side to allow users to use shorthands to
specify resource requirements.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message