hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Merge MR-279 (was Re: Status on MAPREDUCE-279 and plans forward)
Date Wed, 13 Jul 2011 06:52:42 GMT
On Jun 10, 2011, at 11:44 PM, Arun C Murthy wrote:

> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and MR-279.  
> This is a very important patch by Tom which allows to submit jobs to  
> both JT and the new RM - thanks Tom!
> 2. Check-in the 'yarn' subfolder which is just the ResourceManager and  
> NodeManager i.e. compute fabric to trunk.
> 3. Rework the src tree in trunk to look like the MR-279 branch to  
> ensure no changes to trunk are lost - this will ensure separation  
> between the 'classic' framework (JT/TT) and the MR runtime (map/reduce/ 
> sort/shuffle etc.)
> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279  
> branch.
> 5. Change build scripts to reflect the new reality.

Hello,

Another update...

I've been working on 'step 4' outlined above.

Currently the MR 'runtime' in trunk is nearly identical to the MR-279 (including MAPREDUCE-2400,
thanks Tom!) but for the following 4 jiras for which I'm requesting some help from the named
folks to port to trunk:

$ git show 99ab6d89acc2fc722489c317be796c15c3975284
Vinod - MAPREDUCE-2681. Added env vars (stderr/stdout) for hadoop pipes 

$ git show 39a446c12a60041a2740d1a1b99e362a7ea1c8a5
Robert Evans - MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID

$ git show 5ece56360977444ca5799dc06c581e215ff3e10b
Robert Evans - MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient

$ git show 6bddb76f515c3fc178078b688b8d3198b496dba3
Sid_S - MR-2365 in-lieu

We also plan to port HADOOP-6929 to MR-279 (Mahadev).

At the end of this we should have both trunk and MR-279 in sync for the MR runtime. Hopefully
we can get to this by end of this week (optimistically).

----

Once we achieve this we are done with step 1 and step 4, I plan to go with the rest of the
steps outlined above.

Thus, in an ideal scenario I should be able to merge MR-279 into trunk by end of next week!

We have already have been fortunate to receive several contributions from people not sitting
in my aisle and we'd love more! *smile*

I'd love to have more feedback and review bandwidth. Please take a look now - I'm hoping we
can get this into trunk soon!

thanks,
Arun
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message