Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Sun, 15 Oct 2017 03:34:02 +0000 (UTC)
From: "Wangda Tan (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13091743.1501634033000.138.1508038442054@Atlassian.JIRA>
In-Reply-To: <JIRA.13091743.1501634033000@Atlassian.JIRA>
References: <JIRA.13091743.1501634033000@Atlassian.JIRA> <JIRA.13091743.1501634033794@jira-lw-us.apache.org>
Subject: [jira] [Commented] (YARN-6927) Add support for individual resource
 types requests in MapReduce
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sun, 15 Oct 2017 03:34:12 -0000


    [ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205005#comment-16205005 ] 

Wangda Tan commented on YARN-6927:
----------------------------------

[~templedf], since MR only supports specifying individual resource types, I think we can add the support of specifying all types in single field in the future.

Before this patch, user can specify
{code}
yarn.app.mapreduce.am.resource.mb for AM's memory 
yarn.app.mapreduce.am.cpu-vcores for AM's vcores
mapreduce.map/reduce.memory.mb for mapper/reducer's memory 
mapreduce.map/reduce.cpu.vcores for mapper/reducer's vcores. 
{code}

After this patch, user can specify:
{code}
For AM's memory:
yarn.app.mapreduce.am.resource.mb OR 
yarn.app.mapreduce.am.resource.memory-mb = 1024 MB 

For AM's vcores:
yarn.app.mapreduce.am.cpu-vcores OR
yarn.app.mapreduce.am.resource.vcores = 5

For Other AM resources:
yarn.app.mapreduce.am.resource.disk-space = 12 GB

For Mapper/Reducer's Memory:
mapreduce.map/reduce.memory.mb OR 
mapreduce.map/reduce.resource.memory-mb  = 1024 MB

For Mapper/Reducer's vcore:
mapreduce.map/reduce.cpu.vcores OR 
mapreduce.map/reduce.resource.vcores  = 3

For other Mapper/Reducer's resource:
mapreduce.map/reduce.resource.disk-space = 30 GB
{code}

First,
I think we should throw exception when following options are specified:
{code}
yarn.app.mapreduce.am.resource.memory-mb
yarn.app.mapreduce.am.resource.vcores
mapreduce.map/reduce.resource.memory-mb 
mapreduce.map/reduce.resource.vcores
{code}
Because overwriting logic (from the implementation, new options like mapreduce.map/reduce.resource.vcores overwrite the old one. This is very confusing to end users.

Second,
It might be better to support specifying space between number and unit. E.g. 1024MB, 1024 MB are all valid and mean the same thing. currently it doesn't allow adding a space.

Thoughts?

> Add support for individual resource types requests in MapReduce
> ---------------------------------------------------------------
>
>                 Key: YARN-6927
>                 URL: https://issues.apache.org/jira/browse/YARN-6927
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Daniel Templeton
>            Assignee: Gergo Repas
>         Attachments: YARN-6927.000.patch, YARN-6927.001.patch
>
>
> YARN-6504 adds support for resource profiles in MapReduce jobs, but resource profiles don't give users much flexibility in their resource requests.  To satisfy users' needs, MapReduce should also allow users to specify arbitrary resource requests.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org