hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
Date Sun, 15 Oct 2017 03:34:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205005#comment-16205005

Wangda Tan commented on YARN-6927:

[~templedf], since MR only supports specifying individual resource types, I think we can add
the support of specifying all types in single field in the future.

Before this patch, user can specify
yarn.app.mapreduce.am.resource.mb for AM's memory 
yarn.app.mapreduce.am.cpu-vcores for AM's vcores
mapreduce.map/reduce.memory.mb for mapper/reducer's memory 
mapreduce.map/reduce.cpu.vcores for mapper/reducer's vcores. 

After this patch, user can specify:
For AM's memory:
yarn.app.mapreduce.am.resource.mb OR 
yarn.app.mapreduce.am.resource.memory-mb = 1024 MB 

For AM's vcores:
yarn.app.mapreduce.am.cpu-vcores OR
yarn.app.mapreduce.am.resource.vcores = 5

For Other AM resources:
yarn.app.mapreduce.am.resource.disk-space = 12 GB

For Mapper/Reducer's Memory:
mapreduce.map/reduce.memory.mb OR 
mapreduce.map/reduce.resource.memory-mb  = 1024 MB

For Mapper/Reducer's vcore:
mapreduce.map/reduce.cpu.vcores OR 
mapreduce.map/reduce.resource.vcores  = 3

For other Mapper/Reducer's resource:
mapreduce.map/reduce.resource.disk-space = 30 GB

I think we should throw exception when following options are specified:
Because overwriting logic (from the implementation, new options like mapreduce.map/reduce.resource.vcores
overwrite the old one. This is very confusing to end users.

It might be better to support specifying space between number and unit. E.g. 1024MB, 1024
MB are all valid and mean the same thing. currently it doesn't allow adding a space.


> Add support for individual resource types requests in MapReduce
> ---------------------------------------------------------------
>                 Key: YARN-6927
>                 URL: https://issues.apache.org/jira/browse/YARN-6927
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Daniel Templeton
>            Assignee: Gergo Repas
>         Attachments: YARN-6927.000.patch, YARN-6927.001.patch
> YARN-6504 adds support for resource profiles in MapReduce jobs, but resource profiles
don't give users much flexibility in their resource requests.  To satisfy users' needs, MapReduce
should also allow users to specify arbitrary resource requests.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message