hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
Date Sun, 15 Oct 2017 03:34:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205005#comment-16205005
] 

Wangda Tan commented on YARN-6927:
----------------------------------

[~templedf], since MR only supports specifying individual resource types, I think we can add
the support of specifying all types in single field in the future.

Before this patch, user can specify
{code}
yarn.app.mapreduce.am.resource.mb for AM's memory 
yarn.app.mapreduce.am.cpu-vcores for AM's vcores
mapreduce.map/reduce.memory.mb for mapper/reducer's memory 
mapreduce.map/reduce.cpu.vcores for mapper/reducer's vcores. 
{code}

After this patch, user can specify:
{code}
For AM's memory:
yarn.app.mapreduce.am.resource.mb OR 
yarn.app.mapreduce.am.resource.memory-mb = 1024 MB 

For AM's vcores:
yarn.app.mapreduce.am.cpu-vcores OR
yarn.app.mapreduce.am.resource.vcores = 5

For Other AM resources:
yarn.app.mapreduce.am.resource.disk-space = 12 GB

For Mapper/Reducer's Memory:
mapreduce.map/reduce.memory.mb OR 
mapreduce.map/reduce.resource.memory-mb  = 1024 MB

For Mapper/Reducer's vcore:
mapreduce.map/reduce.cpu.vcores OR 
mapreduce.map/reduce.resource.vcores  = 3

For other Mapper/Reducer's resource:
mapreduce.map/reduce.resource.disk-space = 30 GB
{code}

First,
I think we should throw exception when following options are specified:
{code}
yarn.app.mapreduce.am.resource.memory-mb
yarn.app.mapreduce.am.resource.vcores
mapreduce.map/reduce.resource.memory-mb 
mapreduce.map/reduce.resource.vcores
{code}
Because overwriting logic (from the implementation, new options like mapreduce.map/reduce.resource.vcores
overwrite the old one. This is very confusing to end users.

Second,
It might be better to support specifying space between number and unit. E.g. 1024MB, 1024
MB are all valid and mean the same thing. currently it doesn't allow adding a space.

Thoughts?

> Add support for individual resource types requests in MapReduce
> ---------------------------------------------------------------
>
>                 Key: YARN-6927
>                 URL: https://issues.apache.org/jira/browse/YARN-6927
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Daniel Templeton
>            Assignee: Gergo Repas
>         Attachments: YARN-6927.000.patch, YARN-6927.001.patch
>
>
> YARN-6504 adds support for resource profiles in MapReduce jobs, but resource profiles
don't give users much flexibility in their resource requests.  To satisfy users' needs, MapReduce
should also allow users to specify arbitrary resource requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message