Mailing-List: contact general-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@incubator.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CANpmDOeayC5-7EeGVWgdc27Y++dNesiNz_e1znYHn98RW-tLaQ@mail.gmail.com>
References: 
 <CANpmDOeayC5-7EeGVWgdc27Y++dNesiNz_e1znYHn98RW-tLaQ@mail.gmail.com>
Date: Fri, 8 Nov 2013 08:49:06 -0800
Message-ID: 
 <CAHZL8y_WRYWdGC8nHFS4BxhLUGbjH7ciKmZ6z0NePDzxzs5mBg@mail.gmail.com>
Subject: Re: [VOTE] Accept Twill for Incubation
From: Andrei Savu <asavu@apache.org>
To: general@incubator.apache.org
Content-Type: multipart/alternative; boundary=e89a8f642bdc13665104eaad291f

--e89a8f642bdc13665104eaad291f
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

+1 (binding)
On Nov 7, 2013 1:04 PM, "Andreas Neumann" <anew@apache.org> wrote:

> The discussion about the Weave proposal has calmed. As the outcome of the
> discussion, we have chosen a new name for the project, Twill. I would lik=
e
> to call a vote for Twill to become an incubated project.
>
> The proposal is pasted below, and also available at:
> https://wiki.apache.org/incubator/TwillProposal
>
> Let's keep this vote open for three business days, closing the voting on
> Tuesday 11/12.
>
> [ ] +1 Accept Twill into the Incubator
> [ ] +0 Don't care.
> [ ] -1 Don't accept Twill because...
>
> -Andreas.
>
> =3D Abstract =3D
>
> Twill is an abstraction over Apache Hadoop=AE YARN that reduces the
> complexity of developing distributed applications, allowing developers to
> focus more on their business logic.
>
> =3D Proposal =3D
>
> Twill is a set of libraries that reduces the complexity of developing
> distributed applications. It exposes the distributed capabilities of Apac=
he
> Hadoop=AE YARN via a simple and intuitive programming model similar to Ja=
va
> threads. Twill also has built-in capabilities required by many distribute=
d
> applications, such as real-time application logs and metrics collection,
> application lifecycle management, and network service discovery.
>
> =3D Background =3D
>
> Hadoop YARN is a generic cluster resource manager that supports any type =
of
> distributed application. However, YARN=92s interfaces are too low level f=
or
> rapid application development. It requires a great deal of boilerplate co=
de
> even for a simple application, creating a high ramp up cost that can turn
> developers away.
>
> Twill is designed to improve this situation with a programming model that
> makes running distributed applications as easy as running Java threads.
> With the abstraction provided by Twill, applications can be executed in
> process threads during development and unit testing and then be deployed =
to
> a YARN cluster without any modifications.
>
> Twill also has built-in support for real-time application logs and metric=
s
> collection, delegation token renewal, application lifecycle management, a=
nd
> network service discovery. This greatly reduces the pain that developers
> face when developing, debugging, deploying and monitoring distributed
> applications.
>
> Twill is not a replacement for YARN, it=92s a framework that operates on =
top
> of YARN.
>
> =3D Rationale =3D
>
> Developers who write YARN applications typically find themselves
> implementing the same (or similar) boilerplate code over and over again
> for every application. It makes sense to distill this common code into a
> reusable set of libraries that is perpetually maintained and improved by =
a
> diverse community of developers.
>
> Twill=92s simple thread-like programming model will enable many Java
> programmers to develop distributed applications. We believe that this
> simplicity will attract developers who would otherwise be discouraged by
> complexity, and many new use cases will emerge for the usage of YARN.
>
> Incubating Twill as an Apache project makes sense because Twill is a
> framework built on top of YARN, and Twill uses Apache Zookeeper, HDFS,
> Kafka, and other Apache software (see the External Dependencies section).
>
> =3D Current Status =3D
>
> Twill was initially developed at Continuuity under the name of Weave. The
> Weave codebase is currently hosted in a public repository at github.com,
> which will seed the Apache git repository after renaming to Twill.
>
> =3D=3D Meritocracy =3D=3D
>
> Our intent with this incubator proposal is to start building a diverse
> developer community around Twill following the Apache meritocracy model.
> Since Twill was initially developed in early 2013, we have had fast
> adoption and contributions within Continuuity. We are looking forward to
> new contributors. We wish to build a community based on Apache's
> meritocracy principles, working with those who contribute significantly t=
o
> the project and welcoming them to be committers both during the incubatio=
n
> process and beyond.
>
> =3D=3D Community =3D=3D
>
> Twill is currently being used internally at Continuuity and is at the cor=
e
> of our products. We hope to extend our contributor base significantly and
> we will invite all who are interested in simplifying the development of
> distributed applications to participate.
>
> =3D=3D Core Developers =3D=3D
>
> Twill is currently being developed by five engineers at Continuuity:
> Terence Yim, Andreas Neumann, Gary Helmling, Poorna Chandra and Albert
> Shau.
> Terence Yim is an Apache committer for Helix, Andreas is an Apache
> committer and PMC member for Oozie, and Gary Helmling is an Apache
> committer and PMC member for HBase. Poorna Chandra and Albert Shau have
> made many contributions to Twill.
>
> =3D=3D Alignment =3D=3D
>
> The ASF is the natural choice to host the Twill project as its goal of
> encouraging community-driven open source projects fits with our vision fo=
r
> Twill.
>
> Additionally, many other projects with which we are familiar and expect
> Twill to integrate with, such as ZooKeeper, YARN, HDFS, log4j, and others
> mentioned in the External Dependencies section are Apache projects, and
> Twill will benefit by close proximity to them.
>
> =3D Known Risks =3D
>
> =3D=3D Orphaned Products =3D=3D
>
> There is very little risk of Twill being orphaned, as it is a key part of
> Continuuity=92s products. The core Twill developers plan to continue to w=
ork
> on Twill, and Continuuity has funding in place to support their efforts
> going forward.
> Many other Big Data companies can benefit from Twill, and we have already
> received interest from various entities that would like to use and
> contribute to Twill.
>
> =3D=3D Inexperience with Open Source =3D=3D
>
> Several of the core developers have experience with open source
> development. Terence Yim, Andreas Neumann and Gary Helmling are currently
> Apache committers for Helix, Oozie and HBase respectively
> Homogeneous Developers
>
> The current core developers are all Continuuity employees. However, we
> intend to establish a developer community that includes independent and
> corporate contributors. We are encouraging new contributors via our maili=
ng
> lists, public presentations, and personal contacts, and we will continue =
to
> do so. Various entities have already expressed interest in becoming
> involved with Twill.
>
> =3D=3D Reliance on Salaried Developers =3D=3D
>
> Currently, these developers are paid to work on Twill. Once the project h=
as
> built a community, we expect to attract committers, developers and
> community
> other than the current core developers. However, because Continuuity
> products use Twill internally, the reliance on salaried developers is
> unlikely to change, at least in the near term.
>
> =3D=3D Relationships with Other Apache Products =3D=3D
>
> Twill is deeply integrated with Apache projects. Twill uses Apache YARN a=
s
> its underlying resource management and task scheduling system and Apache
> Zookeeper for coordination. In addition, Twill uses Apache HDFS and Apach=
e
> Kafka. A number of other Apache projects are Twill dependencies and are
> listed in the External Dependencies section.
>
> =3D=3D An Excessive Fascination with the Apache Brand =3D=3D
>
> While we respect the reputation of the Apache brand and have no doubt tha=
t
> it will attract contributors and users, our interest is primarily to give
> Twill a solid home as an open source project following an established
> development model. We have also given additional reasons in the Rationale
> and Alignment sections.
>
> =3D Documentation =3D
>
> The current documentation for Weave is at
> https://github.com/continuuity/weave.
> This will be adapted to the new name Twill.
>
> =3D Initial Source =3D
>
> The Weave codebase is currently hosted at
> https://github.com/continuuity/weave. Weave will be renamed to Twill to
> seed the Apache git repository.
>
> =3D External Dependencies =3D
>
> The dependencies all have Apache-compatible licenses:
>  * avro (Apache 2.0)
>  * hadoop (Apache 2.0)
>  * gson (Apache 2.0)
>  * guava-libraries (Apache 2.0)
>  * hbase (Apache 2.0)
>  * hdfs (Apache 2.0)
>  * kafka (Apache 2.0)
>  * netty (Apache 2.0)
>  * snappy-java (Apache 2.0)
>  * yarn (Apache 2.0)
>  * zookeeper (Apache 2.0)
>  * asm (BSD)
>  * junit (EPL v1.0)
>  * logback (EPL v1.0 )
>  * slf4j (MIT)
>
> =3D Cryptography =3D
>
> Twill will depend on secure Hadoop, which can optionally use Kerberos.
>
> =3D Required Resources =3D
>
> =3D=3D Mailing Lists =3D=3D
>
>  * twill-private for private PMC discussions (with moderated subscription=
s)
>  * twill-dev for technical discussions among contributors
>  * twill-commits for notification about commits
>
> =3D=3D Subversion Directory =3D=3D
>
> Git is the preferred source control system: git://git.apache.org/twill
>
> =3D=3D Issue Tracking =3D=3D
>
> JIRA Twill (TWILL)
>
> =3D=3D Other Resources =3D=3D
>
> The existing code already has unit tests, so we would like a Hudson
> instance to run them whenever a new patch is submitted. This can be added
> after project creation.
>
> =3D Initial Committers =3D
>
>  * Terence Yim
>  * Andreas Neumann
>  * Gary Helmling
>  * Poorna Chandra
>  * Albert Shau
>
> =3D Affiliations =3D
>
>  * Terence Yim (Continuuity)
>  * Andreas Neumann (Continuuity)
>  * Gary Helmling (Continuuity)
>  * Poorna Chandra (Continuuity)
>  * Albert Shau (Continuuity)
>
> =3D Sponsors =3D
>
> =3D=3D Champion =3D=3D
>
> Vinod K <vinodkv at apache dot org> (Apache Member)
>
> =3D=3D Nominated Mentors =3D=3D
>
>  * Arun C Murthy <acmurthy at apache dot org>
>  * Tom White <tomwhite at apache dot org>
>  * Patrick Hunt <phunt at apache dot org>
>  * Andrei Savu <asavu at apache dot org>
>
> =3D=3D Sponsoring Entity =3D=3D
>
> We are requesting that the Incubator sponsor this project.
>

--e89a8f642bdc13665104eaad291f--