incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Shaposhnik <...@apache.org>
Subject Re: [PROPOSAL] Weave for Apache Incubator
Date Mon, 04 Nov 2013 07:21:58 GMT
Have you guys considered contributing this code directly to Hadoop?

>From what you are describing it sounds like a developer friendly library
wrapped around YARN (along the lines of kitten:
https://github.com/jwills/kitten).


Thanks,
Roman.

On Tue, Oct 29, 2013 at 12:53 PM, Andreas Neumann <anew@apache.org> wrote:
> I would like to propose Weave, an abstraction over Apache Hadoop® YARN to
> reduce the complexity of developing distributed applications, as an
> Apache Incubator
> podling.
>
> The proposal is included in plain text. I would also like to put this on
> the wiki, but I appear to lack privileges to create pages. What do I need
> to do to get permission?
>
> -Andreas.
>
> Abstract
> ========
>
> Weave is an abstraction over Apache Hadoop® YARN that reduces the
> complexity of developing distributed applications, allowing developers to
> focus more on their business logic.
>
> Proposal
> ========
>
> Weave is a set of libraries that reduces the complexity of developing
> distributed applications. It exposes the distributed capabilities of Apache
> Hadoop® YARN via a simple and intuitive programming model similar to Java
> threads. Weave also has built-in capabilities required by many distributed
> applications, such as real-time application logs and metrics collection,
> application lifecycle management, and network service discovery.
>
> Background
> ==========
>
> Hadoop YARN is a generic cluster resource manager that supports any type of
> distributed application. However, YARN’s interfaces are too low level for
> rapid application development. It requires a great deal of boilerplate code
> even for a simple application, creating a high ramp up cost that can turn
> developers away.
>
> Weave is designed to improve this situation with a programming model that
> makes running distributed applications as easy as running Java threads.
> With the abstraction provided by Weave, applications can be executed in
> process threads during development and unit testing and then be deployed to
> a YARN cluster without any modifications.
>
> Weave also has built-in support for real-time application logs and metrics
> collection, delegation token renewal, application lifecycle management, and
> network service discovery. This greatly reduces the pain that developers
> face when developing, debugging, deploying and monitoring distributed
> applications.
>
> Weave is not a replacement for YARN, it’s a framework that operates on top
> of YARN.
>
> Rationale
> =========
>
> Developers who write YARN applications typically find themselves
> implementing the same (or similar) boilerplate code over and over again for
> every application. It makes sense to distill this common code into a
> reusable set of libraries that is perpetually maintained and improved by a
> diverse community of developers.
>
> Weave’s simple thread-like programming model will enable many Java
> programmers to develop distributed applications. We believe that this
> simplicity will attract developers who would otherwise be discouraged by
> complexity, and many new use cases will emerge for the usage of YARN.
>
> Incubating Weave as an Apache project makes sense because Weave is a
> framework built on top of YARN, and Weave uses Apache Zookeeper, HDFS,
> Kafka, and other Apache software (see the External Dependencies section).
>
> Current Status
> ==============
>
> Weave was initially developed at Continuuity. The Weave codebase is
> currently hosted in a public repository at github.com, which will seed the
> Apache git repository.
>
> Meritocracy
> -----------
> Our intent with this incubator proposal is to start building a diverse
> developer community around Weave following the Apache meritocracy model.
> Since Weave was initially developed in early 2013, we have had fast
> adoption and contributions within Continuuity. We are looking forward to
> new contributors. We wish to build a community based on Apache's
> meritocracy principles, working with those who contribute significantly to
> the project and welcoming them to be committers both during the incubation
> process and beyond.
>
> Community
> ---------
> Weave is currently being used internally at Continuuity and is at the core
> of our products. We hope to extend our contributor base significantly and
> we will invite all who are interested in simplifying the development of
> distributed applications to participate.
>
> Core Developers
> ---------------
> Weave is currently being developed by five engineers at Continuuity:
>  Terence Yim, Andreas Neumann, Gary Helmling, Poorna Chandra and Albert
> Shau.
> Terence Yim is an Apache committer for Helix, Andreas is an Apache
> committer and PMC member for Oozie, and Gary Helmling is an Apache
> committer and PMC member for HBase. Poorna Chandra and Albert Shau have
> made many contributions to Weave.
>
> Alignment
> ---------
> The ASF is the natural choice to host the Weave project as its goal of
> encouraging community-driven open source projects fits with our vision for
> Weave.
>
> Additionally, many other projects with which we are familiar and expect
> Weave to integrate with, such as ZooKeeper, YARN, HDFS, log4j, and others
> mentioned in the External Dependencies section are Apache projects, and
> Weave will benefit by close proximity to them.
>
> Known Risks
> ===========
>
> Orphaned Products
> -----------------
> There is very little risk of Weave being orphaned, as it is a key part of
> Continuuity’s products. The core Weave developers plan to continue to work
> on Weave, and Continuuity has funding in place to support their efforts
> going forward.
> Many other Big Data companies can benefit from Weave, and we have already
> received interest from various entities that would like to use and
> contribute to Weave.
>
> Inexperience with Open Source
> -----------------------------
> Several of the core developers have experience with open source
> development. Terence Yim, Andreas Neumann and Gary Helmling are currently
> Apache committers for Helix, Oozie and HBase respectively
> Homogeneous Developers
>
> The current core developers are all Continuuity employees. However, we
> intend to establish a developer community that includes independent and
> corporate contributors. We are encouraging new contributors via our mailing
> lists, public presentations, and personal contacts, and we will continue to
> do so. Various entities have already expressed interest in becoming
> involved with Weave.
>
> Reliance on Salaried Developers
> -------------------------------
> Currently, these developers are paid to work on Weave. Once the project has
> built a community, we expect to attract committers, developers and
> community other the current core developers. However, because Continuuity
> products use Weave internally, the reliance on salaried developers is
> unlikely to change, at least in the near term.
>
> Relationships with Other Apache Products
> ----------------------------------------
> Weave is deeply integrated with Apache projects. Weave uses Apache YARN as
> its underlying resource management and task scheduling system and Apache
> Zookeeper for coordination. In addition, Weave uses Apache HDFS and Apache
> Kafka. A number of other Apache projects are Weave dependencies and are
> listed in the External Dependencies section.
>
> An Excessive Fascination with the Apache Brand
> ----------------------------------------------
> While we respect the reputation of the Apache brand and have no doubt that
> it will attract contributors and users, our interest is primarily to give
> Weave a solid home as an open source project following an established
> development model. We have also given additional reasons in the Rationale
> and Alignment sections.
>
> Documentation
> =============
>
> https://github.com/continuuity/weave
>
> Initial Source
> ==============
>
> The Weave codebase is currently hosted at
> https://github.com/continuuity/weave.
>
> External Dependencies
> =====================
>
> The dependencies all have Apache-compatible licenses:
> • avro (Apache 2.0)
> • hadoop (Apache 2.0)
> • gson (Apache 2.0)
> • guava-libraries (Apache 2.0)
> • hbase (Apache 2.0)
> • hdfs (Apache 2.0)
> • kafka (Apache 2.0)
> • netty (Apache 2.0)
> • snappy-java (Apache 2.0)
> • yarn (Apache 2.0)
> • zookeeper (Apache 2.0)
> • asm (BSD)
> • junit (EPL v1.0)
> • logback (EPL v1.0 )
> • slf4j (MIT)
>
> Cryptography
> ============
>
> Weave will depend on secure Hadoop, which can optionally use Kerberos.
>
> Required Resources
> ==================
>
> Mailing Lists
> -------------
> - weave-private for private PMC discussions (with moderated subscriptions)
> - weave-dev for technical discussions among contributors
> - weave-commits for notification about commits
>
> Subversion Directory
> --------------------
> Git is the preferred source control system: git://git.apache.org/weave
>
> Issue Tracking
> --------------
> JIRA Weave (WEAVE)
>
> Other Resources
> ---------------
> The existing code already has unit tests, so we would like a Hudson
> instance to run them whenever a new patch is submitted. This can be added
> after project creation.
>
> Initial Committers
> ==================
>
> • Terence Yim
> • Andreas Neumann
> • Gary Helmling
> • Poorna Chandra
> • Albert Shau
>
> Affiliations
> ============
>
> • Terence Yim (Continuuity)
> • Andreas Neumann (Continuuity)
> • Gary Helmling (Continuuity)
> • Poorna Chandra (Continuuity)
> • Albert Shau (Continuuity)
>
> Sponsors
> ========
>
> Champion
> --------
> Vinod K <vinodkv at apache dot org> (Apache Member)
>
> Nominated Mentors
> -----------------
> • Arun C Murthy <acmurthy at apache dot org>
> • Tom White <tomwhite at apache dot org>
> • Patrick Hunt <phunt at apache dot org>
>
> Sponsoring Entity
> -----------------
> We are requesting that the Incubator sponsor this project.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message