hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: [ANNOUNCEMENT] Yahoo focusing on Apache Hadoop, discontinuing "The Yahoo Distribution of Hadoop"
Date Thu, 07 Apr 2011 22:52:44 GMT

On Feb 14, 2011, at 1:34 PM, Arun C Murthy wrote:
> As the final installment in this process, I've started a discussion on
> us contributing a re-factor of Map-Reduce in https://issues.apache.org/jira/browse/MAPREDUCE-279
> .

Hi Folks,

We wanted to share our thoughts around the co-development of the  
NextGen MapReduce branch (Jira MR-279), maintaining the branch-0.20- 
security and merging the work on the security branch with trunk.   
We've concluded that it does not make sense for us to port a very  
small subset of the work from the branch-0.20-security to the Hadoop  
mainline.  The JIRAs we don't plan to port all effect areas of the  
mainline that are going to be replaced by work in the NextGen  
MapReduce branch (http://svn.apache.org/viewvc/hadoop/mapreduce/branches/MR-279/ 

We've been working on the NextGen MapReduce branch (MAPREDUCE-279)  
within Apache for a while now and are excited about it's progress.  We  
think that this branch will be a huge improvement in scalability,  
performance and functionality.  We are now confident that we can get  
it ready for release in in the next few months.  We believe that the  
next major release of Apache Hadoop we will test at Yahoo will include  
the work in this branch and we are committed to merging the NextGen  
branch into the mainline after the PMC approves the merge.

Meanwhile, we have continued to find and fix bugs on branch-0.20- 
security and have been working to port that work into the Hadoop  
mainline.  Most of this work is done and we've also brought all the  
patches in from our github branch into apache subversion, so that it  
is easy for everyone to see the work remaining.  What we've found is  
that some of the work in branch-0.20-security is in code sections that  
have been completely replaced / refactored in the NextGen MapReduce  
branch.  Since we are committed to the NextGen branch, we don't think  
there is any upside in porting this code into portions of mainline we  
expect to discard. All of these JIRAs will be fixed in the NextGen  
MapReduce branch and through there ultimately in trunk (assuming the  
PMC approves the merge).

So at this point it is our intent to not port the JIRAs listed above  
to trunk, but to wait until we merge NextGen into trunk to resolve  
these issues there.  If you are interested in seeing these issues  
ported to mainline, let us know.  We are happy to help review your  
patches and explain context to anyone who is interested in doing this  

Arun and Eric

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message