incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Neumann <a...@apache.org>
Subject Re: [PROPOSAL] Weave for Apache Incubator
Date Tue, 05 Nov 2013 23:06:24 GMT
Hi Andrei, thanks for offering, we are happy about any help we can get.


On Mon, Nov 4, 2013 at 10:43 AM, Andrei Savu <asavu@apache.org> wrote:

> IMO it's a good idea to keep this a separate project - the same way we have
> Curator for ZooKeeper.
>
> If you are looking for one more mentor I'm happy to help.
>
> -- Andrei Savu
>
> On Sun, Nov 3, 2013 at 11:21 PM, Roman Shaposhnik <rvs@apache.org> wrote:
>
> > Have you guys considered contributing this code directly to Hadoop?
> >
> > From what you are describing it sounds like a developer friendly library
> > wrapped around YARN (along the lines of kitten:
> > https://github.com/jwills/kitten).
> >
> >
> > Thanks,
> > Roman.
> >
> > On Tue, Oct 29, 2013 at 12:53 PM, Andreas Neumann <anew@apache.org>
> wrote:
> > > I would like to propose Weave, an abstraction over Apache Hadoop® YARN
> to
> > > reduce the complexity of developing distributed applications, as an
> > > Apache Incubator
> > > podling.
> > >
> > > The proposal is included in plain text. I would also like to put this
> on
> > > the wiki, but I appear to lack privileges to create pages. What do I
> need
> > > to do to get permission?
> > >
> > > -Andreas.
> > >
> > > Abstract
> > > ========
> > >
> > > Weave is an abstraction over Apache Hadoop® YARN that reduces the
> > > complexity of developing distributed applications, allowing developers
> to
> > > focus more on their business logic.
> > >
> > > Proposal
> > > ========
> > >
> > > Weave is a set of libraries that reduces the complexity of developing
> > > distributed applications. It exposes the distributed capabilities of
> > Apache
> > > Hadoop® YARN via a simple and intuitive programming model similar to
> Java
> > > threads. Weave also has built-in capabilities required by many
> > distributed
> > > applications, such as real-time application logs and metrics
> collection,
> > > application lifecycle management, and network service discovery.
> > >
> > > Background
> > > ==========
> > >
> > > Hadoop YARN is a generic cluster resource manager that supports any
> type
> > of
> > > distributed application. However, YARN’s interfaces are too low level
> for
> > > rapid application development. It requires a great deal of boilerplate
> > code
> > > even for a simple application, creating a high ramp up cost that can
> turn
> > > developers away.
> > >
> > > Weave is designed to improve this situation with a programming model
> that
> > > makes running distributed applications as easy as running Java threads.
> > > With the abstraction provided by Weave, applications can be executed in
> > > process threads during development and unit testing and then be
> deployed
> > to
> > > a YARN cluster without any modifications.
> > >
> > > Weave also has built-in support for real-time application logs and
> > metrics
> > > collection, delegation token renewal, application lifecycle management,
> > and
> > > network service discovery. This greatly reduces the pain that
> developers
> > > face when developing, debugging, deploying and monitoring distributed
> > > applications.
> > >
> > > Weave is not a replacement for YARN, it’s a framework that operates on
> > top
> > > of YARN.
> > >
> > > Rationale
> > > =========
> > >
> > > Developers who write YARN applications typically find themselves
> > > implementing the same (or similar) boilerplate code over and over again
> > for
> > > every application. It makes sense to distill this common code into a
> > > reusable set of libraries that is perpetually maintained and improved
> by
> > a
> > > diverse community of developers.
> > >
> > > Weave’s simple thread-like programming model will enable many Java
> > > programmers to develop distributed applications. We believe that this
> > > simplicity will attract developers who would otherwise be discouraged
> by
> > > complexity, and many new use cases will emerge for the usage of YARN.
> > >
> > > Incubating Weave as an Apache project makes sense because Weave is a
> > > framework built on top of YARN, and Weave uses Apache Zookeeper, HDFS,
> > > Kafka, and other Apache software (see the External Dependencies
> section).
> > >
> > > Current Status
> > > ==============
> > >
> > > Weave was initially developed at Continuuity. The Weave codebase is
> > > currently hosted in a public repository at github.com, which will seed
> > the
> > > Apache git repository.
> > >
> > > Meritocracy
> > > -----------
> > > Our intent with this incubator proposal is to start building a diverse
> > > developer community around Weave following the Apache meritocracy
> model.
> > > Since Weave was initially developed in early 2013, we have had fast
> > > adoption and contributions within Continuuity. We are looking forward
> to
> > > new contributors. We wish to build a community based on Apache's
> > > meritocracy principles, working with those who contribute significantly
> > to
> > > the project and welcoming them to be committers both during the
> > incubation
> > > process and beyond.
> > >
> > > Community
> > > ---------
> > > Weave is currently being used internally at Continuuity and is at the
> > core
> > > of our products. We hope to extend our contributor base significantly
> and
> > > we will invite all who are interested in simplifying the development of
> > > distributed applications to participate.
> > >
> > > Core Developers
> > > ---------------
> > > Weave is currently being developed by five engineers at Continuuity:
> > >  Terence Yim, Andreas Neumann, Gary Helmling, Poorna Chandra and Albert
> > > Shau.
> > > Terence Yim is an Apache committer for Helix, Andreas is an Apache
> > > committer and PMC member for Oozie, and Gary Helmling is an Apache
> > > committer and PMC member for HBase. Poorna Chandra and Albert Shau have
> > > made many contributions to Weave.
> > >
> > > Alignment
> > > ---------
> > > The ASF is the natural choice to host the Weave project as its goal of
> > > encouraging community-driven open source projects fits with our vision
> > for
> > > Weave.
> > >
> > > Additionally, many other projects with which we are familiar and expect
> > > Weave to integrate with, such as ZooKeeper, YARN, HDFS, log4j, and
> others
> > > mentioned in the External Dependencies section are Apache projects, and
> > > Weave will benefit by close proximity to them.
> > >
> > > Known Risks
> > > ===========
> > >
> > > Orphaned Products
> > > -----------------
> > > There is very little risk of Weave being orphaned, as it is a key part
> of
> > > Continuuity’s products. The core Weave developers plan to continue to
> > work
> > > on Weave, and Continuuity has funding in place to support their efforts
> > > going forward.
> > > Many other Big Data companies can benefit from Weave, and we have
> already
> > > received interest from various entities that would like to use and
> > > contribute to Weave.
> > >
> > > Inexperience with Open Source
> > > -----------------------------
> > > Several of the core developers have experience with open source
> > > development. Terence Yim, Andreas Neumann and Gary Helmling are
> currently
> > > Apache committers for Helix, Oozie and HBase respectively
> > > Homogeneous Developers
> > >
> > > The current core developers are all Continuuity employees. However, we
> > > intend to establish a developer community that includes independent and
> > > corporate contributors. We are encouraging new contributors via our
> > mailing
> > > lists, public presentations, and personal contacts, and we will
> continue
> > to
> > > do so. Various entities have already expressed interest in becoming
> > > involved with Weave.
> > >
> > > Reliance on Salaried Developers
> > > -------------------------------
> > > Currently, these developers are paid to work on Weave. Once the project
> > has
> > > built a community, we expect to attract committers, developers and
> > > community other the current core developers. However, because
> Continuuity
> > > products use Weave internally, the reliance on salaried developers is
> > > unlikely to change, at least in the near term.
> > >
> > > Relationships with Other Apache Products
> > > ----------------------------------------
> > > Weave is deeply integrated with Apache projects. Weave uses Apache YARN
> > as
> > > its underlying resource management and task scheduling system and
> Apache
> > > Zookeeper for coordination. In addition, Weave uses Apache HDFS and
> > Apache
> > > Kafka. A number of other Apache projects are Weave dependencies and are
> > > listed in the External Dependencies section.
> > >
> > > An Excessive Fascination with the Apache Brand
> > > ----------------------------------------------
> > > While we respect the reputation of the Apache brand and have no doubt
> > that
> > > it will attract contributors and users, our interest is primarily to
> give
> > > Weave a solid home as an open source project following an established
> > > development model. We have also given additional reasons in the
> Rationale
> > > and Alignment sections.
> > >
> > > Documentation
> > > =============
> > >
> > > https://github.com/continuuity/weave
> > >
> > > Initial Source
> > > ==============
> > >
> > > The Weave codebase is currently hosted at
> > > https://github.com/continuuity/weave.
> > >
> > > External Dependencies
> > > =====================
> > >
> > > The dependencies all have Apache-compatible licenses:
> > > • avro (Apache 2.0)
> > > • hadoop (Apache 2.0)
> > > • gson (Apache 2.0)
> > > • guava-libraries (Apache 2.0)
> > > • hbase (Apache 2.0)
> > > • hdfs (Apache 2.0)
> > > • kafka (Apache 2.0)
> > > • netty (Apache 2.0)
> > > • snappy-java (Apache 2.0)
> > > • yarn (Apache 2.0)
> > > • zookeeper (Apache 2.0)
> > > • asm (BSD)
> > > • junit (EPL v1.0)
> > > • logback (EPL v1.0 )
> > > • slf4j (MIT)
> > >
> > > Cryptography
> > > ============
> > >
> > > Weave will depend on secure Hadoop, which can optionally use Kerberos.
> > >
> > > Required Resources
> > > ==================
> > >
> > > Mailing Lists
> > > -------------
> > > - weave-private for private PMC discussions (with moderated
> > subscriptions)
> > > - weave-dev for technical discussions among contributors
> > > - weave-commits for notification about commits
> > >
> > > Subversion Directory
> > > --------------------
> > > Git is the preferred source control system: git://git.apache.org/weave
> > >
> > > Issue Tracking
> > > --------------
> > > JIRA Weave (WEAVE)
> > >
> > > Other Resources
> > > ---------------
> > > The existing code already has unit tests, so we would like a Hudson
> > > instance to run them whenever a new patch is submitted. This can be
> added
> > > after project creation.
> > >
> > > Initial Committers
> > > ==================
> > >
> > > • Terence Yim
> > > • Andreas Neumann
> > > • Gary Helmling
> > > • Poorna Chandra
> > > • Albert Shau
> > >
> > > Affiliations
> > > ============
> > >
> > > • Terence Yim (Continuuity)
> > > • Andreas Neumann (Continuuity)
> > > • Gary Helmling (Continuuity)
> > > • Poorna Chandra (Continuuity)
> > > • Albert Shau (Continuuity)
> > >
> > > Sponsors
> > > ========
> > >
> > > Champion
> > > --------
> > > Vinod K <vinodkv at apache dot org> (Apache Member)
> > >
> > > Nominated Mentors
> > > -----------------
> > > • Arun C Murthy <acmurthy at apache dot org>
> > > • Tom White <tomwhite at apache dot org>
> > > • Patrick Hunt <phunt at apache dot org>
> > >
> > > Sponsoring Entity
> > > -----------------
> > > We are requesting that the Incubator sponsor this project.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message