Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of eli@cloudera.com designates
 74.125.83.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <0C112F45-3130-4093-B0F2-2D1F02B8C84C@jpl.nasa.gov>
References: <0C112F45-3130-4093-B0F2-2D1F02B8C84C@jpl.nasa.gov>
Date: Wed, 29 Aug 2012 11:41:29 -0700
Message-ID: 
 <CAPn_vTt=D3RtgSMa1KN5OnLXXV5MP9cXj9j+gzdPcQ28ZVBoHA@mail.gmail.com>
Subject: Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and
 disband Hadoop umbrella project
From: Eli Collins <eli@cloudera.com>
To: general@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

Thanks for writing up a proposal Chris.

I think it makes sense to have Common live in HDFS at least for now,
since it's at the bottom of the stack / dependency chain and it's code
is the most intertwined with common, and, per Arun, we tend to work on
common stuff more than MR people. The HDFS project is really a lot
more than HDFS, eg has all the hadoop commands, non-HDFS file system
source, etc but that seems like an OK starting point. We need to
figure out the committers and PMC though since the goal is to just
have the HDFS community (vs the current Hadoop people) but the project
will contain non-HDFS stuff. I'd like to hear from the current Hadoop
committers and PMC members that mostly work on MR and YARN - are you
guys OK losing your current privileges on the HDFS repo? Otherwise we
haven't made much progress (ie HDFS still has multiple communities).

We also need to address the areas where it's not so cut and dry, eg
where there is a single Hadoop project:
- The Hadoop trademark, assume this lives in the HDFS project if Common does?
- The user community, eg the users lists that we *just* merged, shall
we still keep one list?
- We should move the global stuff like "how to get started" docs to
Bigtop, which can point to individual projects resources
- Hadoop 1.x is is maintenance mode, though it still actively gets
patches so we need to consider it. The surgery necessary to split v1
Hadoop is probably not suitable for a sustaining release and not worth
it at this point in the lifetime of this branch. I assume the HDFS
project will then host the Hadoop 1.x branches?  This implies only
members of the HDFS project can commit and release.

Thanks,
Eli

On Tue, Aug 28, 2012 at 7:33 PM, Mattmann, Chris A (388J)
<chris.a.mattmann@jpl.nasa.gov> wrote:
> [decided to minimize traffic and to simply put this in one thread]
>
> Hi Guys,
>
> See the recent discussion on these threads:
>
> YARN as its own Hadoop "sub project": http://s.apache.org/WW1
> Maintain a single committer list for the Hadoop project: http://s.apache.org/Owx
>
> ...and just pay attention to the Hadoop project over the last 3-4 years. It's operating
> as a single project, that's masking separate communities that themselves are really
> separate ASF projects.
>
> At the ASF, this has been a problem area called "umbrella" projects and over the years,
> all I've seen from them is wasted bandwidth, artificial barriers and the inventions of
> new ways to perform process mongering and to reduce the fun in developing software
> at this fantastic foundation.
>
> I've talked about umbrella projects enough. We've diverted conversation enough.
> Enough people have tried to act like there is some technical mumbo jumbo that is
> preventing the eventual act of higher power that I myself hope comes should these
> discussions prove unfruitful through normal means.
>
> *these. are. separate. projects.*
> *there.are.not.blocker.issues.from.spinning.out.these.projects.as.their.own.communities*
>
> In this email: http://s.apache.org/rSm
>
> And in the 2 subsequent follow ons in that thread, I've outlined a process that I'll copy
> through below for splitting these projects into their own TLPs:
>
> -----snip
> Process:
>
> 0. [DISCUSS] thread for <TLP name> in which you talk about #1 and #2 below, potentially draft resolution too.
>
> 1. Decide on an initial set of *PMC* members. I urge each new TLP to adopt PMC==C. See reasons I've
> already discussed.
>
> 2. Decide on a chair. Try not to VOTE for this explicitly, see if can be discussed and consensus
> can be reached (just a thought experiment). VOTE if necessary.
>
> 3. [VOTE] thread for <TLP name>
>
> 4. Create Project:
>   a. paste resolution from #0 to board@ or;
>   b. go to general@incubator and start new Incubator project.
>
> 5. infrastructure set up.
>    MLs moving; new UNIX groups; website setup;
>    SVN setup like this:
>
> svn copy -m "MR TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool MR name>; or
> svn copy -m "YARN TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool YARN name>; or
> svn copy -m "HDFS TLP." https://svn.apache.org/repos/asf/hadoop/ https://svn.apache.org/repos/asf/<insert cool HDFS name>
>
> After all 3 have been created run:
>
> svn remove -m "Remove Hadoop umbrella TLP. Split into separate projects." https://svn.apache.org/repos/asf/hadoop
>
> 6. (TLPs if 4a; Incubator podling if 4b;) proceed, collaborate, operate as distinct communities, and try to solve the code duplication/dependency
> issues from there.
>
> 7. If 4b; then graduate as TLP from Incubator.
>
> -----snip
>
> So that's my proposal.
>
> Thanks guys.
>
> Cheers,
> Chris
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Chris Mattmann, Ph.D.
> Senior Computer Scientist
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 171-266B, Mailstop: 171-246
> Email: chris.a.mattmann@nasa.gov
> WWW:   http://sunset.usc.edu/~mattmann/
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> Adjunct Assistant Professor, Computer Science Department
> University of Southern California, Los Angeles, CA 90089 USA
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>