hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
Date Mon, 27 Apr 2015 23:37:39 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14515853#comment-14515853

Wangda Tan commented on MAPREDUCE-6304:

bq. Well my view should be to support them as i feel in some cases we might feel like mapper
can run in any node but may be reducer requires more mem so may be clients might require it
to be run in high mem nodes (may be some constraint on mem for the nodes) or some other node
That's correct, so you agree to add mapper/reducer node-label-expression?

bq. Well, IIUC then ApplicationSubmissionContext.getNodeLabelExpression() is for mapper/reducer
and AM's node label expression is set in ApplicationSubmissionContext.getAMContainerResourceRequest.
I could cross verify the same with code in ApplicationMasterService (line number 495) &
RMAppManager (line number 378) and so did not get what you meant by the above statement, correct
me if my understanding is wrong.
What I meant is:
When job.label = x, and don't set am/mapper/reducer label, all containers should get allocated
on x 
When job.label = x and am.label = y, don't set mapper/reducer label, am will get allocated
on y (overwritten x) but mapper/reducer should get allocated on x.
When job.label = x, am.label=y, mapper.label=z, am will get allocated on y, mapper will get
allocated on z (overwrite x), and reducer get allocated on x.
This should be existing behavior. And mapper/reducer's label should be added to ResourceRequest
(may need modify {{RMContainerRequestor}}) instead of AplicationSubmissionContext.

> Specifying node labels when submitting MR jobs
> ----------------------------------------------
>                 Key: MAPREDUCE-6304
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Jian Fang
>            Assignee: Naganarasimha G R
>             Fix For: 2.8.0
>         Attachments: MAPREDUCE-6304.20150410-1.patch, MAPREDUCE-6304.20150411-1.patch
> Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels
when submitting MR jobs.

This message was sent by Atlassian JIRA

View raw message