hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
Date Mon, 13 Apr 2015 18:40:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14492818#comment-14492818

Jian Fang commented on MAPREDUCE-6304:

I mean hadoop could provide a new mechanism such as a decorator for the ApplicationSubmissionContext.
When the method submitApplication() in class ClientRMService is called, hadoop decorates the
ApplicationSubmissionContext before it calls the following line. For example, manipulates
the amLabelExpression. 

          System.currentTimeMillis(), user);

Hadoop could provide a default decorator that does nothing. But users could override the default
decorator in yarn-site.xml by a new configuration parameter, for example, "yarn.app.submission.context.decorator.class".

This new mechanism is not directly related to the change you are making, but it is more generic
so that the platform providers could update the ApplicationSubmissionContext in their own
ways. Once we have such a new mechanism in place, you do not really need to add anything new
to your label code for my use case. Instead, the custom logic will be included in the custom
decorator provided by the platform provider. For example, we could provide a decorator to
update amLabelExpression in ApplicationSubmissionContext. Other fields of ApplicationSubmissionContext
could be changed as well to meet user's needs.

> Specifying node labels when submitting MR jobs
> ----------------------------------------------
>                 Key: MAPREDUCE-6304
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Jian Fang
>            Assignee: Naganarasimha G R
>             Fix For: 2.8.0
>         Attachments: MAPREDUCE-6304.20150410-1.patch, MAPREDUCE-6304.20150411-1.patch
> Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels
when submitting MR jobs.

This message was sent by Atlassian JIRA

View raw message