hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-279) Map-Reduce 2.0
Date Fri, 18 Mar 2011 14:35:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13008452#comment-13008452

Arun C Murthy commented on MAPREDUCE-279:

Thanks for your f/b Tom.

bq. I wonder if it would be easier not to move the src/java/org/apache/hadoop/mapred(uce)
trees at this stage.

The main issue is the dependency chain - currently the mr-client depends purely on apis in
yarn package. In the alternate proposal (which we considered) mr-client would need to depend
on yarn and src/java for the runtime. 

The current scheme is both more modular and enforces discipline by ensuring that the MapReduce
runtime (map, sort, shuffle, merge, reduce) cannot, even accidentally, start relying on classes
in the server package i.e. JT/TT etc. This also seems like the right end-state for the project.

Also, as you pointed out, the changes to classes in src/java/org/apache/hadoop/mapred(uce)
are very minor and the 'svn mv' is both well documented (MR-279_MR_files_to_move.txt, MR-279.sh)
and straight-forward.


bq. MAPREDUCE-1638 is highly relevant for this work

Thanks! MAPREDUCE-1638 is very relevant. MAPREDUCE-279 already has some of the changes you
proposed there i.e. keeping server classes in a separate source structure from the implementation
classes - we should collaborate both on trunk and on the MR-279 branch to ensure consistency.
I'm happy to merge if necessary. 

> Map-Reduce 2.0
> --------------
>                 Key: MAPREDUCE-279
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.23.0
>         Attachments: MR-279.patch, MR-279.patch, MR-279.sh, MR-279_MR_files_to_move.txt
> Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component
that manages the application execution. 

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message