hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: [DISCUSS] Spin out MR, HDFS and YARN as their own TLPs and disband Hadoop umbrella project
Date Fri, 31 Aug 2012 16:58:54 GMT
The problem there is that YARN depends on Common, and MapReduce depends on
YARN, so we would either have a circular dependency or we would have to
split off MapRedcue too.


On 8/31/12 11:54 AM, "Eli Collins" <eli@cloudera.com> wrote:

>How about a proposal to just spin YARN off as a TLP?  Rationale:
>1. YARN started as a separate project and has a more independent
>community than Common/HDFS/MR (per below these communities do not
>divide at sub-project boundaries) that appears to want to be even more
>2. YARN is technically much easier to separate from the rest of the
>code base (than separating Common and HDFS for example). Separating it
>out will also help accelerate other efforts like MR2 support for
>Apache Mesos.
>3. It side steps a number of thorny issues (how to handle branch-1,
>how to handle what Hadoop is wrt enforcing trademark, who to remove
>people from the Hadoop PMC, etc) that haven't been addressed in any of
>these proposals.
>4. It's a proof point - if you can't make the case for YARN then
>there's no way we're going to make a case for splitting the other
>projects (this thread).
>Ie this doesn't have to be an all-or-nothing proposition for all
>sub-projects, since the communities don't fall on sub-project
>On Tue, Aug 28, 2012 at 7:33 PM, Mattmann, Chris A (388J)
><chris.a.mattmann@jpl.nasa.gov> wrote:
>> [decided to minimize traffic and to simply put this in one thread]
>> Hi Guys,
>> See the recent discussion on these threads:
>> YARN as its own Hadoop "sub project": http://s.apache.org/WW1
>> Maintain a single committer list for the Hadoop project:
>> ...and just pay attention to the Hadoop project over the last 3-4
>>years. It's operating
>> as a single project, that's masking separate communities that
>>themselves are really
>> separate ASF projects.
>> At the ASF, this has been a problem area called "umbrella" projects and
>>over the years,
>> all I've seen from them is wasted bandwidth, artificial barriers and
>>the inventions of
>> new ways to perform process mongering and to reduce the fun in
>>developing software
>> at this fantastic foundation.
>> I've talked about umbrella projects enough. We've diverted conversation
>> Enough people have tried to act like there is some technical mumbo
>>jumbo that is
>> preventing the eventual act of higher power that I myself hope comes
>>should these
>> discussions prove unfruitful through normal means.
>> *these. are. separate. projects.*
>> In this email: http://s.apache.org/rSm
>> And in the 2 subsequent follow ons in that thread, I've outlined a
>>process that I'll copy
>> through below for splitting these projects into their own TLPs:
>> -----snip
>> Process:
>> 0. [DISCUSS] thread for <TLP name> in which you talk about #1 and #2
>>below, potentially draft resolution too.
>> 1. Decide on an initial set of *PMC* members. I urge each new TLP to
>>adopt PMC==C. See reasons I've
>> already discussed.
>> 2. Decide on a chair. Try not to VOTE for this explicitly, see if can
>>be discussed and consensus
>> can be reached (just a thought experiment). VOTE if necessary.
>> 3. [VOTE] thread for <TLP name>
>> 4. Create Project:
>>   a. paste resolution from #0 to board@ or;
>>   b. go to general@incubator and start new Incubator project.
>> 5. infrastructure set up.
>>    MLs moving; new UNIX groups; website setup;
>>    SVN setup like this:
>> svn copy -m "MR TLP." https://svn.apache.org/repos/asf/hadoop/
>>https://svn.apache.org/repos/asf/<insert cool MR name>; or
>> svn copy -m "YARN TLP." https://svn.apache.org/repos/asf/hadoop/
>>https://svn.apache.org/repos/asf/<insert cool YARN name>; or
>> svn copy -m "HDFS TLP." https://svn.apache.org/repos/asf/hadoop/
>>https://svn.apache.org/repos/asf/<insert cool HDFS name>
>> After all 3 have been created run:
>> svn remove -m "Remove Hadoop umbrella TLP. Split into separate
>>projects." https://svn.apache.org/repos/asf/hadoop
>> 6. (TLPs if 4a; Incubator podling if 4b;) proceed, collaborate, operate
>>as distinct communities, and try to solve the code duplication/dependency
>> issues from there.
>> 7. If 4b; then graduate as TLP from Incubator.
>> -----snip
>> So that's my proposal.
>> Thanks guys.
>> Cheers,
>> Chris
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW:   http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

View raw message