Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "<general@hadoop.apache.org>" <general@hadoop.apache.org>
Subject: Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and
 disband Hadoop umbrella project
Thread-Topic: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and
 disband Hadoop umbrella project
Thread-Index: AQHNhY6+cKAg7NaI+EWRM79fMSTKeJdwnQgAgACvYAA=
Date: Wed, 29 Aug 2012 14:14:42 +0000
Message-ID: <7F5019B7-CFB1-4A12-958E-F6745875F39D@jpl.nasa.gov>
References: <0C112F45-3130-4093-B0F2-2D1F02B8C84C@jpl.nasa.gov>
 <CAJs-t7PsYUx+VJrhr5cDkOMQ3e8vvuKH7AkUq5kcT0C7meJLRg@mail.gmail.com>
In-Reply-To: 
 <CAJs-t7PsYUx+VJrhr5cDkOMQ3e8vvuKH7AkUq5kcT0C7meJLRg@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-ID: <A1D41CA7489C4746933DFD3756597517@ad.jpl>
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Hi Alejandro,

On Aug 28, 2012, at 8:50 PM, Alejandro Abdelnur wrote:

> Chris, thanks for initiating the discussion.

No probs!

>=20
> IMO a pre-requisite to this is to figure out how we'll handle the followi=
ng:
>=20

To be honest, I don't think any of the below are prereqs. They are technica=
l
issues that can be dealt with post facto of just SVN copy'ing hadoop as it=
=20
stands today per my SVN commands into each of the new TLPs and then
using that as a starting point for doing the below, as part of the natural =
evolution
of the project code.=20

That being said, if I had to guess what the TLPs would do to address the be=
low
once they are created:

> * Where does common stuff lives?

This usually happens over time and depending on how often things release,=20
and other things cited else-threads, and else-discussions over the past yea=
rs
in Hadoop. You guys clearly have a good handle on things like this.=20

I would just encourage the subsequent TLPs to not worry about doing everyth=
ing
perfectly and to realize that if you start out with the same code base, you=
 can selectively
and then iteratively just make things more clean, refactored, and the answe=
r to questions
like this will happen naturally during that evolution.

> * What are the public interfaces of each project (towards the other proje=
cts)?

This is something that each distinct community can answer once they are boo=
tstrapped
as TLPs. You can decide what portion of the code is really under charter an=
d then work
as a community to figure this out. Sorry I can't be more specific than that=
.

> * How do we do development/releases? In tandem? Separate?

In tandem across communities never really works. Releases should occur sepa=
rately, per
community and TLP, on their own schedule. Code that depends on other projec=
ts either
has to wait for those communities/TLPs/projects to fix things, or add new f=
eatures, or=20
whatever, or insulate, and keep the fixes locally in your project's SVN unt=
il those fixes
can be pushed upstream, and included in the other communities releases, etc=
.

Ask yourself this. If you guys have a dependency on e.g., Tomcat, and there=
 is some critical
bug or new feature you want in Tomcat, how would you deal with that? I woul=
d posit the same
way that you could deal with this situation. Keep the fix to Tomcat locally=
 in your project;=20
work to get that fix upstream and included in some subsequent Tomcat releas=
e, etc.

> How this
> will work in practice, currently we are constantly tweaking things
> inter-projects, sometimes in the same JIRAs, sometimes in follow up
> JIRAs.

Technically you are doing that that, but community wise, it's not working o=
ut, and hasn't
really been working for years. I've been around Hadoop since its inception =
(I was a Nutch
committer before Hadoop existed), and though it's been hugely successful, a=
nd really=20
awesome and super great (congrats, everyone, BTW!), the community issues ha=
ve always
cropped up b/c it's one big huge umbrella project and that doesn't work at =
Apache.

Cheers,
Chris

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++