hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
Date Fri, 15 May 2015 18:04:00 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545896#comment-14545896

Naganarasimha G R commented on MAPREDUCE-6304:

Hi [~wangda]
I have already taken care of this 
* {{JOB_NODE_LABEL_EXP}} is set to the Application submission context and yarn takes care
of setting only when ResourceRequest is ANY
* {{AM_NODE_LABEL_EXP}} is set to the AM's Resource request which is of Any type always. 
* {{MAP_NODE_LABEL_EXP}} & {{REDUCE_NODE_LABEL_EXP}} are added to the resource request
only when its of {{ANY}} type in RMContainerRequestor.addContainerReq(ContainerRequest). Test
case is also present for it.

My question was in most cases container is assigned to either Node Local or Rack local, So
in these cases container cannot be assigned to a Node satisfying the Node Label exp. So my
suggestion was based on boolean config flag can we decide whether to add node local and rack
local request or not. for example expose  ENABLE_ANY_RESOURCE_REQUEST_ONLY config based on
which node local and rack local request will not be set.

> Specifying node labels when submitting MR jobs
> ----------------------------------------------
>                 Key: MAPREDUCE-6304
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Jian Fang
>            Assignee: Naganarasimha G R
>              Labels: mapreduce
>         Attachments: MAPREDUCE-6304.20150410-1.patch, MAPREDUCE-6304.20150411-1.patch,
MAPREDUCE-6304.20150501-1.patch, MAPREDUCE-6304.20150510-1.patch, MAPREDUCE-6304.20150511-1.patch,
> Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels
when submitting MR jobs.

This message was sent by Atlassian JIRA

View raw message