hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Hammerbacher (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-279) Map-Reduce 2.0
Date Mon, 21 Feb 2011 23:16:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997599#comment-12997599

Jeff Hammerbacher commented on MAPREDUCE-279:

Hey Arun,

As long as that evolution is happening in a branch, that seems totally reasonable to me. When
it comes time to migrate the code into trunk, I hope for the same end state as Matei: I think
the resource management system should be a separate project from MapReduce so that each system
can evolve and release separately. When we have more clients than just MapReduce for the resource
manager, we'll want those new clients to evolve as separate projects rather than all living
under the Apache Hadoop umbrella. Now seems like an excellent time to facilitate that end

More specifically, in an ideal world, we'd have four separate projects here: Common (probably
folded into Guava or Apache Commons), HDFS, Yahoo! Cluster Manager (Resource Manager + Node
Manager), and MapReduce (the ApplicationMaster for MapReduce, I guess). Then, if someone wanted
to write Pregel to run against the Cluster Manager, they could implement their own ApplicationMaster
in a separate project. Similarly, if someone wanted to run MapReduce against a different cluster
manager, that would be simple. More practically, we have the opportunity to get the Cluster
Manager project started up as a separate ASF project once it has gestated in a branch here
for a bit. Are there any technical barriers to making that happen?

I'm a huge fan of this work, and having watched a number of ASF projects evolve over the past
several years, I suspect that a small, focused project dedicated to cluster resource management
will have the best chance of moving quickly.


> Map-Reduce 2.0
> --------------
>                 Key: MAPREDUCE-279
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.23.0
> Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component
that manages the application execution. 

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message