Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: 216.145.54.171 is neither permitted
 nor denied by domain of eric14@yahoo-inc.com)
From: Eric Baldeschwieler <eric14@yahoo-inc.com>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: Thinking about the next hadoop mainline release
Date: Fri, 17 Jun 2011 00:17:56 -0700
Message-Id: <EFF679DA-26AB-4250-8F59-A2330D0E3E5C@yahoo-inc.com>
To: general@hadoop.apache.org
Mime-Version: 1.0 (Apple Message framework v1084)

Hi Folks,

I'd like to start a conversation on mainline planning and the next =
release of Apache Hadoop beyond 0.22.

The Yahoo! Hadoop team has been working hard to complete several big =
Hadoop projects, including:

- HDFS Federation [HDFS-1052]
 - Already merged into trunk

- Next Generation Map-Reduce [MR-279]
 - Passing most tests now and discussing merging into trunk

- The merging of our previous work on Hadoop with security into mainline =
[http://yhoo.it/i9Ww8W]
 - This is mostly done, but owen and others are doing a scrub to close =
out the remaining issues

All of these projects are now reaching a place where we would like to =
combine them with the good work already in 0.22 and put out a new apache =
release, perhaps 0.23.  We think the best way to accomplish that is to =
finish the merge in the next few weeks and then cut a release from =
trunk.

Yahoo stands ready to help us (the Apache Hadoop Community) turn this =
new release into a stable release by running it through its 9 month test =
and burn in process.  The result of that will be another stable release =
such as 0.18, 0.20 or 0.20.203 (hadoop with security).  We have Yahoo!s =
support for this substantial investment because this new release will =
have a great combination of new features for small and very large sites =
alike:
 - New Write Pipeline - HBase support [also in 0.21 & 0.22]
 - Federation - Scale up to larger clusters and the ability to =
experiment with new namenode approaches
 - Next Gen MapReduce - Scaleup, performance improvements, ability to =
experiment with new processing frameworks

I think this effort will produce a great new Apache Hadoop release for =
the community.  I'm starting this thread to collect feedback and =
hopefully folks' endorsement for merging in MR-279 and putting together =
this new release.  Feedback please?

Thanks,

E14