hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: Status on MAPREDUCE-279 and plans forward
Date Sat, 11 Jun 2011 08:10:32 GMT

On Jun 11, 2011, at 6:28 AM, Tom White wrote:

> Hi Arun,
>
> Great to hear that MR2 is nearly ready for trunk.
>
> MR2 is being built using Maven, so step 5 is about getting Maven to
> build all of MR I presume? The svn unsplit (HADOOP-7106) and
> Mavenization work in HADOOP-6671 should make this approach more
> tractable. Or were you thinking that there would be a hybrid Ant/Maven
> build during a transition period?
>

Good point Tom. I'm thinking of having a hybrid ant/maven transition  
period to allow both to proceed independently.

> Also, for step 3, what did you think of the directory layout I
> suggested in http://bit.ly/mr-279-maven-layout?
>

Makes sense, maybe we should stage that too. Lets continue that  
discussion on jira.

thanks!
Arun

> Cheers,
> Tom
>
> On Fri, Jun 10, 2011 at 12:44 PM, Arun C Murthy <acm@yahoo-inc.com>  
> wrote:
>> Another heads up.
>>
>> Things are going well, we have done scale testing (350 nodes) and  
>> continue
>> to do more.
>>
>> We think we are nearly ready to merge MR-279 into trunk (yaay!).
>>
>> A proposal for plan forward:
>>
>> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and  
>> MR-279. This
>> is a very important patch by Tom which allows to submit jobs to  
>> both JT and
>> the new RM - thanks Tom!
>> 2. Check-in the 'yarn' subfolder which is just the ResourceManager  
>> and
>> NodeManager i.e. compute fabric to trunk.
>> 3. Rework the src tree in trunk to look like the MR-279 branch to  
>> ensure no
>> changes to trunk are lost - this will ensure separation between the
>> 'classic' framework (JT/TT) and the MR runtime (map/reduce/sort/ 
>> shuffle
>> etc.)
>> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279  
>> branch.
>> 5. Change build scripts to reflect the new reality.
>>
>> In terms of timelines, I'm hoping we can get this done in the  
>> coming 2-3
>> weeks.
>>
>> Thoughts?
>>
>> thanks,
>> Arun
>>


Mime
View raw message