hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tom White <...@cloudera.com>
Subject Re: Status on MAPREDUCE-279 and plans forward
Date Sat, 11 Jun 2011 00:58:22 GMT
Hi Arun,

Great to hear that MR2 is nearly ready for trunk.

MR2 is being built using Maven, so step 5 is about getting Maven to
build all of MR I presume? The svn unsplit (HADOOP-7106) and
Mavenization work in HADOOP-6671 should make this approach more
tractable. Or were you thinking that there would be a hybrid Ant/Maven
build during a transition period?

Also, for step 3, what did you think of the directory layout I
suggested in http://bit.ly/mr-279-maven-layout?


On Fri, Jun 10, 2011 at 12:44 PM, Arun C Murthy <acm@yahoo-inc.com> wrote:
> Another heads up.
> Things are going well, we have done scale testing (350 nodes) and continue
> to do more.
> We think we are nearly ready to merge MR-279 into trunk (yaay!).
> A proposal for plan forward:
> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and MR-279. This
> is a very important patch by Tom which allows to submit jobs to both JT and
> the new RM - thanks Tom!
> 2. Check-in the 'yarn' subfolder which is just the ResourceManager and
> NodeManager i.e. compute fabric to trunk.
> 3. Rework the src tree in trunk to look like the MR-279 branch to ensure no
> changes to trunk are lost - this will ensure separation between the
> 'classic' framework (JT/TT) and the MR runtime (map/reduce/sort/shuffle
> etc.)
> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279 branch.
> 5. Change build scripts to reflect the new reality.
> In terms of timelines, I'm hoping we can get this done in the coming 2-3
> weeks.
> Thoughts?
> thanks,
> Arun

View raw message