hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jian Fang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
Date Wed, 08 Apr 2015 19:48:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485884#comment-14485884

Jian Fang commented on MAPREDUCE-6304:

I understand your point from on-premise cluster perspective. However, it is not very practical
to manage mapred-site.xml or queue files for users if hadoop is a service in cloud. As a hadoop
developer, you should consider both on-premise hadoop cluster and hadoop in cloud. 

There are many many users for a hadoop cloud service. Usually they launch their own hadoop
clusters in cloud and control their own queue files or mapred-site.xml.  Some of them even
run their hadoop jobs on their own gateways that the hadoop platform provider does not have
access to. But the hadoop service provider may still want to have a mechanism to set up some
global labels for all users to improve their user experiences. For example, a failure of an
application master on a spot instance due to the termination of a spot instance will cause
more trouble than a failure of one MR task. These types of settings most likely can only be
done by hadoop cloud service providers based on their deep knowledge in their own cloud services.

Or could hadoop provide a mechanism for hadoop providers to extend so that you only need to
specify the labels in YarnRunner in Vanilla hadoop?  

> Specifying node labels when submitting MR jobs
> ----------------------------------------------
>                 Key: MAPREDUCE-6304
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Jian Fang
>            Assignee: Naganarasimha G R
> Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify node labels
when submitting MR jobs.

This message was sent by Atlassian JIRA

View raw message