hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6504) Add support for resource profiles in MapReduce
Date Mon, 25 Sep 2017 10:39:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-6504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178832#comment-16178832

Daniel Templeton commented on YARN-6504:

The vast majority of users still only ask for 1 core and 1GB RAM, so the scope of the problem
is limited.  I don't see where a partial solution actually solves anything.  Let's take it
case by case:

* Users who only use CPU and memory: there's no practical difference between specifying CPU
and memory, and specifying a profile and then overriding one (or both!) of the values.
* Users who use other resource types: if a user's job depends on something other than CPU
and memory, letting them only override CPU and memory doesn't save them from having to retest
with profiles.

The main driving reason for having the override capability is that job's which don't ask for
a profile get the default profile and need to be able to ignore it.  What if we get rid of
the default profile?  I don't think it's actually very useful.  It let's the admin set the
default resource request for the cluster, but you can do that for each of the application
frameworks already, and I don't think it makes much sense to have a default that is ignorant
of the application to which it is applied.  I also really don't like that we're creating two
sets of configurations to control the same thing.  If we take away the default, the override
capability isn't useful anymore.  If you ask for a profile, you get the profile.  If not,
you get the resources you ask for.  If you didn't ask for any, you get the usual defaults
for your application framework.

> Add support for resource profiles in MapReduce
> ----------------------------------------------
>                 Key: YARN-6504
>                 URL: https://issues.apache.org/jira/browse/YARN-6504
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: YARN-6504-YARN-3926.001.patch, YARN-6504.YARN-3926.002.patch, YARN-6504.YARN-3926.003.patch

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message