Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted
 nor denied by domain of amberry@yahoo-inc.com)
From: Adam Berry <amberry@yahoo-inc.com>
To: "general@hadoop.apache.org" <general@hadoop.apache.org>
Date: Tue, 6 Nov 2012 09:19:21 -0800
Subject: Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Thread-Topic: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Thread-Index: Ac28QtxTWp8Ar84RQkWUsc24bMnZAw==
Message-ID: <B076495A-C463-4B05-AFBF-D79DBDC75FF3@yahoo-inc.com>
References: <D5D101D8-E951-497A-81E8-B2887A853F31@yahoo-inc.com>
 <FF2B4ABC27D8A544B67850810EA7A8D9026958F3@sara-exch-02.ka.sara.nl>
 <474E4097-7DD6-475C-812F-6ED591B65C73@yahoo-inc.com>
In-Reply-To: <474E4097-7DD6-475C-812F-6ED591B65C73@yahoo-inc.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Just to keep everyone up to date over here as well, the vote has started ov=
er on the general@incubator.apache.org list.

Cheers,
Adam


On Oct 17, 2012, at 5:06 PM, Adam Berry wrote:

> Hi Evert,
>=20
> that sounds interesting and certainly useful. If you could contribute tha=
t would be great. I'll check back in with you in a couple of days.
>=20
> Cheers,
> Adam
>=20
> On Oct 17, 2012, at 4:52 PM, Evert Lammerts wrote:
>=20
>> Hi Adam, thanks for the initiative, this is good news for us and our use=
rs as well - right now I'm advising ant targets for submitting MR jobs from=
 Eclipse. I'll have a look to see whether / how we can contribute.
>>=20
>> Best,
>> Evert
>>=20
>> ________________________________________
>> From: Adam Berry [amberry@yahoo-inc.com]
>> Sent: Wednesday, October 17, 2012 10:01 PM
>> To: general@hadoop.apache.org; general@incubator.apache.org
>> Subject: [DISCUSS] [PROPOSAL] Hadoop Development Tools
>>=20
>> Hello,
>>=20
>> Below is a proposal for a new incubator project. This idea came out of a=
nd had strong support on the general hadoop list, see the thread at http://=
mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser.
>>=20
>> We are looking for feedback, and to see who else would be interested in =
contributing to this effort as a committer, as well as an additional mentor=
.
>>=20
>> Cheers,
>> Adam Berry
>>=20
>>=20
>> Hadoop Development Tools Proposal
>>=20
>> I=92d like to propose the Hadoop Development Tools, a set of extensions =
to the Eclipse IDE to support developing against Apache Hadoop technologies=
.
>>=20
>> =3D HDT (Hadoop Development Tools) =3D
>>=20
>> =3D=3D Abstract =3D=3D
>> Tools to support developing applications that use Apache Hadoop from wit=
hin Eclipse.
>>=20
>> =3D=3D Proposal =3D=3D
>> Hadoop Development Tools are a set of extensions to Eclipse providing su=
pport for creating, launching and debugging distributed applications, as we=
ll as interacting with HDFS filesystems. This work will build on the existi=
ng Map Reduce Tools present in the Apache Hadoop project.
>>=20
>> =3D=3D Background =3D=3D
>> Map Reduce Tools have existed as part of contrib for Apache Hadoop. Unfo=
rtunately they are source tied to a single version of Hadoop, and developme=
nt has stalled, with little movement past the Hadoop 0.20 line.
>>=20
>> =3D=3D Rationale =3D=3D
>> Support for newer versions of Hadoop from within Eclipse is regularly ra=
ised on the Hadoop mailing lists, so there is a clear need to drive these t=
ools forward. Development tools generally are worked on separate from the t=
arget tools/platform, separating the tools out will allow for supporting mu=
ltiple versions, so a developer could work with a heterogeneous environment=
.
>>=20
>> =3D=3D Initial Goals =3D=3D
>> * Give the tools project a home of its own.
>> * Port current MapReduce tools feature set to all current release lines =
of Hadoop in a single Eclipse install.
>> * Documentation and tutorials for all features.
>> * Publish Eclipse update site, and join Eclipse marketplace listing.
>> * Establish release cycle that combines support for Hadoop and Eclipse r=
elease cycles.
>> * Look to build support for YARN, MRUnit and possibly other Hadoop-relat=
ed projects.
>>=20
>> =3D=3D Current Status =3D=3D
>> The source for the current MapReduceTools lives in the contrib section o=
f the Hadoop source. In its current implementation it is tied to the versio=
n of Hadoop against which it is compiled. The layout and API that it was de=
veloped with means that it can only be used with the 0.20 or 1.0 Hadoop rel=
eases, the new layout and YARN api introduced with the 0.23 and 2.0 lines a=
re not supported.
>>=20
>>=20
>> =3D=3D=3D Meritocracy =3D=3D=3D
>> Several people and companies have already expressed an interest in contr=
ibuting to this project, and we hope to attract additional interest during =
the proposal discussion. We plan to invest and support a meritocracy that a=
ttracts, invites, and supports newcomers to build a vibrant and  diverse co=
mmunity.
>>=20
>> =3D=3D=3D Community =3D=3D=3D
>> The target community is developers who are working developing Map/Reduce=
 applications against Hadoop. Given the success of Hadoop the target group =
is likely to be quite large. Separation from the Hadoop community would mak=
e it easier to support multiple versions of hadoop, as well as merging the =
release cycles of Hadoop and Eclipse to provide predictable iteration and i=
mprovement in the toolset.
>>=20
>> =3D=3D=3D Core Developers =3D=3D=3D
>> The initial list of developers includes people experienced with developi=
ng against the Eclipse platform.
>> * Adam Berry (amberry at yahoo-inc dot com)
>> * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
>>=20
>> =3D=3D=3D Alignment =3D=3D=3D
>> Hadoop Development Tools aligns with both Hadoop and Eclipse. Hadoop as =
the platform for the development target, and Eclipse as the IDE platform us=
ed as the base for the tools.
>>=20
>> =3D=3D Known Risks =3D=3D
>>=20
>> =3D=3D=3D Orphaned Products =3D=3D=3D
>>=20
>> =3D=3D=3D Inexperience with Open Source =3D=3D=3D
>> Adam Berry has experience of the Eclipse open source community, and has =
been building familiarity with the Apache processes through patches to the =
existing source.
>>=20
>> =3D=3D=3D Reliance on Salaried Developers =3D=3D=3D
>> Hadoop Development Tools will be developed with a mix of salaried and vo=
lunteer time.
>>=20
>> =3D=3D=3D Relationships with Other Apache Projects =3D=3D=3D
>> Hadoop Development Tools is closely related to Apache Hadoop.
>>=20
>> =3D=3D=3D An Excessive Fascination with the Apache Brand =3D=3D=3D
>> Given the success of Hadoop and associated projects, Apache is the natur=
al place for the Hadoop Development Tools. Chris Mattman suggested the Apac=
he Incubator as appropriate on the Hadoop general mailing list following th=
e success that MRUnit had taking the path from Hadoop contrib to an Apache =
top level project.
>>=20
>> =3D=3D Documentation =3D=3D
>> Documentation for the current tools can be found at http://wiki.apache.o=
rg/hadoop/EclipsePlugIn
>>=20
>> =3D=3D Initial Source =3D=3D
>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-pro=
ject/src/contrib/eclipse-plugin/
>>=20
>> =3D=3D  Source and Intellectual Property Submission Plan =3D=3D
>> The source, and any suggested initial patches, are already hosted either=
 in Apache=92s Subversion or JIRA.
>>=20
>> =3D=3D  External Dependencies =3D=3D
>> Eclipse Platform
>> Eclipse Java Development Tools
>>=20
>> =3D=3D  Cryptography =3D=3D
>> Hadoop Development Tools likely does not fall into this area.
>>=20
>> =3D=3D  Required Resources =3D=3D
>> =3D=3D=3D Mailing lists =3D=3D=3D
>> * hdt-dev
>> * hdt-commits
>> * hdt-user
>>=20
>> =3D=3D=3D Subversion Directory =3D=3D=3D
>> * https://svn.apache.org/repos/asf/incubator/hdt
>>=20
>> =3D=3D=3D Issue Tracking =3D=3D=3D
>> * JIRA Hadoop Development Tools (HDT)
>>=20
>> =3D=3D=3D Other Resources =3D=3D=3D
>> * Jenkins/Hudson for builds and test running.
>>=20
>> =3D=3D Initial Committers =3D=3D
>> * Adam Berry (amberry at yahoo-inc dot com)
>> * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
>>=20
>> =3D=3D Affiliations =3D=3D
>> * Adam Berry - Yahoo!
>> * Jeffrey Zemerick - Mountain Fog
>>=20
>> =3D=3D Sponsors =3D=3D
>> =3D=3D=3D Champion =3D=3D=3D
>> Chris Douglas
>>=20
>> =3D=3D=3D Nominated Mentors =3D=3D=3D
>> Chris Douglas
>> Chris Mattman
>>=20
>> =3D=3D=3D Sponsoring Entity =3D=3D=3D
>> Incubator PMC
>=20