hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Merge MR-279 (was Re: Status on MAPREDUCE-279 and plans forward)
Date Fri, 29 Jul 2011 06:11:36 GMT
So, we are getting very close to the merge. Please take time to review and provide f/b.

thanks,
Arun

On Jul 27, 2011, at 9:46 AM, Mahadev Konar wrote:

> Just an update. 
> 
> Some good news. I have a patch for MR-279 to start working with trunk/common and trunk/hdfs
(some minor fixes). I am facing some issues with secuirty which should be sorted out by EOD
today.
> We should be ready to start the process of merging soon.
> 
> thanks
> mahadev
> 
> On Jul 12, 2011, at 11:52 PM, Arun C Murthy wrote:
> 
>> On Jun 10, 2011, at 11:44 PM, Arun C Murthy wrote:
>> 
>>> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and MR-279.  
>>> This is a very important patch by Tom which allows to submit jobs to  
>>> both JT and the new RM - thanks Tom!
>>> 2. Check-in the 'yarn' subfolder which is just the ResourceManager and  
>>> NodeManager i.e. compute fabric to trunk.
>>> 3. Rework the src tree in trunk to look like the MR-279 branch to  
>>> ensure no changes to trunk are lost - this will ensure separation  
>>> between the 'classic' framework (JT/TT) and the MR runtime (map/reduce/ 
>>> sort/shuffle etc.)
>>> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279  
>>> branch.
>>> 5. Change build scripts to reflect the new reality.
>> 
>> Hello,
>> 
>> Another update...
>> 
>> I've been working on 'step 4' outlined above.
>> 
>> Currently the MR 'runtime' in trunk is nearly identical to the MR-279 (including
MAPREDUCE-2400, thanks Tom!) but for the following 4 jiras for which I'm requesting some help
from the named folks to port to trunk:
>> 
>> $ git show 99ab6d89acc2fc722489c317be796c15c3975284
>> Vinod - MAPREDUCE-2681. Added env vars (stderr/stdout) for hadoop pipes 
>> 
>> $ git show 39a446c12a60041a2740d1a1b99e362a7ea1c8a5
>> Robert Evans - MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID
>> 
>> $ git show 5ece56360977444ca5799dc06c581e215ff3e10b
>> Robert Evans - MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient
>> 
>> $ git show 6bddb76f515c3fc178078b688b8d3198b496dba3
>> Sid_S - MR-2365 in-lieu
>> 
>> We also plan to port HADOOP-6929 to MR-279 (Mahadev).
>> 
>> At the end of this we should have both trunk and MR-279 in sync for the MR runtime.
Hopefully we can get to this by end of this week (optimistically).
>> 
>> ----
>> 
>> Once we achieve this we are done with step 1 and step 4, I plan to go with the rest
of the steps outlined above.
>> 
>> Thus, in an ideal scenario I should be able to merge MR-279 into trunk by end of
next week!
>> 
>> We have already have been fortunate to receive several contributions from people
not sitting in my aisle and we'd love more! *smile*
>> 
>> I'd love to have more feedback and review bandwidth. Please take a look now - I'm
hoping we can get this into trunk soon!
>> 
>> thanks,
>> Arun
> 


Mime
View raw message