incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Neumann <a...@apache.org>
Subject Re: [PROPOSAL] Weave for Apache Incubator
Date Wed, 30 Oct 2013 05:53:07 GMT
Pardon my ignorance, I just looked it up and realized that warp and weft
are indeed related to weaving, so they might work.
I do have the impression, though, that most people would associate Warp
with the speed of light and not with weaving.

-Andreas.

On Wed, Oct 30, 2013 at 1:41 AM, Andreas Neumann <anew@apache.org> wrote:

> Thanks for pointing out these similarities; we were not aware of Commons
> Weaver. Given that Weaver is a sub-project of Commons, would the similarity
> be tolerable? Also, since Weave and Wave are pronounced quite differently,
> I am hoping that they are perceived as different enough.
>
> The name Weave is motivated from the name of YARN - it takes the
> complexity out of yarn by "weaving" it into a simple pattern. Warp and Weft
> don't really convey this meaning.
>
> If the concern about the name similarity is really strong, we will try to
> find another name, and we are open to suggestions. But please do also
> consider the motivation for this naming.
>
> Thanks
>
>
>
> On Tue, Oct 29, 2013 at 7:54 PM, sebb <sebbaz@gmail.com> wrote:
>
>> In which case, maybe consider the related words:
>>
>> Apache Warp
>> Apache Weft
>>
>> Just a thought.
>>
>> On 29 October 2013 22:14, Upayavira <uv@odoko.co.uk> wrote:
>> > And Apache Wave too (which is what I first saw before I read the title
>> > more carefully).
>> >
>> > Upayavira
>> >
>> > On Tue, Oct 29, 2013, at 09:12 PM, Matt Benson wrote:
>> >> Hi,
>> >>   I am concerned about potential confusion with Apache Commons Weaver
>> >>   [1].
>> >>
>> >> Matt
>> >>
>> >> [1] https://commons.apache.org/proper/commons-weaver/
>> >>
>> >>
>> >> On Tue, Oct 29, 2013 at 2:53 PM, Andreas Neumann <anew@apache.org>
>> wrote:
>> >>
>> >> > I would like to propose Weave, an abstraction over Apache Hadoop®
>> YARN to
>> >> > reduce the complexity of developing distributed applications, as an
>> >> > Apache Incubator
>> >> > podling.
>> >> >
>> >> > The proposal is included in plain text. I would also like to put
>> this on
>> >> > the wiki, but I appear to lack privileges to create pages. What do
I
>> need
>> >> > to do to get permission?
>> >> >
>> >> > -Andreas.
>> >> >
>> >> > Abstract
>> >> > ========
>> >> >
>> >> > Weave is an abstraction over Apache Hadoop® YARN that reduces the
>> >> > complexity of developing distributed applications, allowing
>> developers to
>> >> > focus more on their business logic.
>> >> >
>> >> > Proposal
>> >> > ========
>> >> >
>> >> > Weave is a set of libraries that reduces the complexity of developing
>> >> > distributed applications. It exposes the distributed capabilities of
>> Apache
>> >> > Hadoop® YARN via a simple and intuitive programming model similar
to
>> Java
>> >> > threads. Weave also has built-in capabilities required by many
>> distributed
>> >> > applications, such as real-time application logs and metrics
>> collection,
>> >> > application lifecycle management, and network service discovery.
>> >> >
>> >> > Background
>> >> > ==========
>> >> >
>> >> > Hadoop YARN is a generic cluster resource manager that supports any
>> type of
>> >> > distributed application. However, YARN’s interfaces are too low
>> level for
>> >> > rapid application development. It requires a great deal of
>> boilerplate code
>> >> > even for a simple application, creating a high ramp up cost that can
>> turn
>> >> > developers away.
>> >> >
>> >> > Weave is designed to improve this situation with a programming model
>> that
>> >> > makes running distributed applications as easy as running Java
>> threads.
>> >> > With the abstraction provided by Weave, applications can be executed
>> in
>> >> > process threads during development and unit testing and then be
>> deployed to
>> >> > a YARN cluster without any modifications.
>> >> >
>> >> > Weave also has built-in support for real-time application logs and
>> metrics
>> >> > collection, delegation token renewal, application lifecycle
>> management, and
>> >> > network service discovery. This greatly reduces the pain that
>> developers
>> >> > face when developing, debugging, deploying and monitoring distributed
>> >> > applications.
>> >> >
>> >> > Weave is not a replacement for YARN, it’s a framework that operates
>> on top
>> >> > of YARN.
>> >> >
>> >> > Rationale
>> >> > =========
>> >> >
>> >> > Developers who write YARN applications typically find themselves
>> >> > implementing the same (or similar) boilerplate code over and over
>> again for
>> >> > every application. It makes sense to distill this common code into
a
>> >> > reusable set of libraries that is perpetually maintained and
>> improved by a
>> >> > diverse community of developers.
>> >> >
>> >> > Weave’s simple thread-like programming model will enable many Java
>> >> > programmers to develop distributed applications. We believe that this
>> >> > simplicity will attract developers who would otherwise be
>> discouraged by
>> >> > complexity, and many new use cases will emerge for the usage of YARN.
>> >> >
>> >> > Incubating Weave as an Apache project makes sense because Weave is
a
>> >> > framework built on top of YARN, and Weave uses Apache Zookeeper,
>> HDFS,
>> >> > Kafka, and other Apache software (see the External Dependencies
>> section).
>> >> >
>> >> > Current Status
>> >> > ==============
>> >> >
>> >> > Weave was initially developed at Continuuity. The Weave codebase is
>> >> > currently hosted in a public repository at github.com, which will
>> seed the
>> >> > Apache git repository.
>> >> >
>> >> > Meritocracy
>> >> > -----------
>> >> > Our intent with this incubator proposal is to start building a
>> diverse
>> >> > developer community around Weave following the Apache meritocracy
>> model.
>> >> > Since Weave was initially developed in early 2013, we have had fast
>> >> > adoption and contributions within Continuuity. We are looking
>> forward to
>> >> > new contributors. We wish to build a community based on Apache's
>> >> > meritocracy principles, working with those who contribute
>> significantly to
>> >> > the project and welcoming them to be committers both during the
>> incubation
>> >> > process and beyond.
>> >> >
>> >> > Community
>> >> > ---------
>> >> > Weave is currently being used internally at Continuuity and is at
>> the core
>> >> > of our products. We hope to extend our contributor base
>> significantly and
>> >> > we will invite all who are interested in simplifying the development
>> of
>> >> > distributed applications to participate.
>> >> >
>> >> > Core Developers
>> >> > ---------------
>> >> > Weave is currently being developed by five engineers at Continuuity:
>> >> >  Terence Yim, Andreas Neumann, Gary Helmling, Poorna Chandra and
>> Albert
>> >> > Shau.
>> >> > Terence Yim is an Apache committer for Helix, Andreas is an Apache
>> >> > committer and PMC member for Oozie, and Gary Helmling is an Apache
>> >> > committer and PMC member for HBase. Poorna Chandra and Albert Shau
>> have
>> >> > made many contributions to Weave.
>> >> >
>> >> > Alignment
>> >> > ---------
>> >> > The ASF is the natural choice to host the Weave project as its goal
>> of
>> >> > encouraging community-driven open source projects fits with our
>> vision for
>> >> > Weave.
>> >> >
>> >> > Additionally, many other projects with which we are familiar and
>> expect
>> >> > Weave to integrate with, such as ZooKeeper, YARN, HDFS, log4j, and
>> others
>> >> > mentioned in the External Dependencies section are Apache projects,
>> and
>> >> > Weave will benefit by close proximity to them.
>> >> >
>> >> > Known Risks
>> >> > ===========
>> >> >
>> >> > Orphaned Products
>> >> > -----------------
>> >> > There is very little risk of Weave being orphaned, as it is a key
>> part of
>> >> > Continuuity’s products. The core Weave developers plan to continue
>> to work
>> >> > on Weave, and Continuuity has funding in place to support their
>> efforts
>> >> > going forward.
>> >> > Many other Big Data companies can benefit from Weave, and we have
>> already
>> >> > received interest from various entities that would like to use and
>> >> > contribute to Weave.
>> >> >
>> >> > Inexperience with Open Source
>> >> > -----------------------------
>> >> > Several of the core developers have experience with open source
>> >> > development. Terence Yim, Andreas Neumann and Gary Helmling are
>> currently
>> >> > Apache committers for Helix, Oozie and HBase respectively
>> >> > Homogeneous Developers
>> >> >
>> >> > The current core developers are all Continuuity employees. However,
>> we
>> >> > intend to establish a developer community that includes independent
>> and
>> >> > corporate contributors. We are encouraging new contributors via our
>> mailing
>> >> > lists, public presentations, and personal contacts, and we will
>> continue to
>> >> > do so. Various entities have already expressed interest in becoming
>> >> > involved with Weave.
>> >> >
>> >> > Reliance on Salaried Developers
>> >> > -------------------------------
>> >> > Currently, these developers are paid to work on Weave. Once the
>> project has
>> >> > built a community, we expect to attract committers, developers and
>> >> > community other the current core developers. However, because
>> Continuuity
>> >> > products use Weave internally, the reliance on salaried developers
is
>> >> > unlikely to change, at least in the near term.
>> >> >
>> >> > Relationships with Other Apache Products
>> >> > ----------------------------------------
>> >> > Weave is deeply integrated with Apache projects. Weave uses Apache
>> YARN as
>> >> > its underlying resource management and task scheduling system and
>> Apache
>> >> > Zookeeper for coordination. In addition, Weave uses Apache HDFS and
>> Apache
>> >> > Kafka. A number of other Apache projects are Weave dependencies and
>> are
>> >> > listed in the External Dependencies section.
>> >> >
>> >> > An Excessive Fascination with the Apache Brand
>> >> > ----------------------------------------------
>> >> > While we respect the reputation of the Apache brand and have no
>> doubt that
>> >> > it will attract contributors and users, our interest is primarily to
>> give
>> >> > Weave a solid home as an open source project following an established
>> >> > development model. We have also given additional reasons in the
>> Rationale
>> >> > and Alignment sections.
>> >> >
>> >> > Documentation
>> >> > =============
>> >> >
>> >> > https://github.com/continuuity/weave
>> >> >
>> >> > Initial Source
>> >> > ==============
>> >> >
>> >> > The Weave codebase is currently hosted at
>> >> > https://github.com/continuuity/weave.
>> >> >
>> >> > External Dependencies
>> >> > =====================
>> >> >
>> >> > The dependencies all have Apache-compatible licenses:
>> >> > • avro (Apache 2.0)
>> >> > • hadoop (Apache 2.0)
>> >> > • gson (Apache 2.0)
>> >> > • guava-libraries (Apache 2.0)
>> >> > • hbase (Apache 2.0)
>> >> > • hdfs (Apache 2.0)
>> >> > • kafka (Apache 2.0)
>> >> > • netty (Apache 2.0)
>> >> > • snappy-java (Apache 2.0)
>> >> > • yarn (Apache 2.0)
>> >> > • zookeeper (Apache 2.0)
>> >> > • asm (BSD)
>> >> > • junit (EPL v1.0)
>> >> > • logback (EPL v1.0 )
>> >> > • slf4j (MIT)
>> >> >
>> >> > Cryptography
>> >> > ============
>> >> >
>> >> > Weave will depend on secure Hadoop, which can optionally use
>> Kerberos.
>> >> >
>> >> > Required Resources
>> >> > ==================
>> >> >
>> >> > Mailing Lists
>> >> > -------------
>> >> > - weave-private for private PMC discussions (with moderated
>> subscriptions)
>> >> > - weave-dev for technical discussions among contributors
>> >> > - weave-commits for notification about commits
>> >> >
>> >> > Subversion Directory
>> >> > --------------------
>> >> > Git is the preferred source control system: git://
>> git.apache.org/weave
>> >> >
>> >> > Issue Tracking
>> >> > --------------
>> >> > JIRA Weave (WEAVE)
>> >> >
>> >> > Other Resources
>> >> > ---------------
>> >> > The existing code already has unit tests, so we would like a Hudson
>> >> > instance to run them whenever a new patch is submitted. This can be
>> added
>> >> > after project creation.
>> >> >
>> >> > Initial Committers
>> >> > ==================
>> >> >
>> >> > • Terence Yim
>> >> > • Andreas Neumann
>> >> > • Gary Helmling
>> >> > • Poorna Chandra
>> >> > • Albert Shau
>> >> >
>> >> > Affiliations
>> >> > ============
>> >> >
>> >> > • Terence Yim (Continuuity)
>> >> > • Andreas Neumann (Continuuity)
>> >> > • Gary Helmling (Continuuity)
>> >> > • Poorna Chandra (Continuuity)
>> >> > • Albert Shau (Continuuity)
>> >> >
>> >> > Sponsors
>> >> > ========
>> >> >
>> >> > Champion
>> >> > --------
>> >> > Vinod K <vinodkv at apache dot org> (Apache Member)
>> >> >
>> >> > Nominated Mentors
>> >> > -----------------
>> >> > • Arun C Murthy <acmurthy at apache dot org>
>> >> > • Tom White <tomwhite at apache dot org>
>> >> > • Patrick Hunt <phunt at apache dot org>
>> >> >
>> >> > Sponsoring Entity
>> >> > -----------------
>> >> > We are requesting that the Incubator sponsor this project.
>> >> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> > For additional commands, e-mail: general-help@incubator.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message