incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Donald Whytock <dwhyt...@apache.org>
Subject Re: [PROPOSAL] Weave for Apache Incubator
Date Wed, 30 Oct 2013 17:56:48 GMT
Knit, crochet, macrame?

Though, honestly, pulling lower-level components together into higher-level
ones sounds a little like granny squares...

Don


On Wed, Oct 30, 2013 at 1:53 AM, Andreas Neumann <anew@apache.org> wrote:

> Pardon my ignorance, I just looked it up and realized that warp and weft
> are indeed related to weaving, so they might work.
> I do have the impression, though, that most people would associate Warp
> with the speed of light and not with weaving.
>
> -Andreas.
>
> On Wed, Oct 30, 2013 at 1:41 AM, Andreas Neumann <anew@apache.org> wrote:
>
> > Thanks for pointing out these similarities; we were not aware of Commons
> > Weaver. Given that Weaver is a sub-project of Commons, would the
> similarity
> > be tolerable? Also, since Weave and Wave are pronounced quite
> differently,
> > I am hoping that they are perceived as different enough.
> >
> > The name Weave is motivated from the name of YARN - it takes the
> > complexity out of yarn by "weaving" it into a simple pattern. Warp and
> Weft
> > don't really convey this meaning.
> >
> > If the concern about the name similarity is really strong, we will try to
> > find another name, and we are open to suggestions. But please do also
> > consider the motivation for this naming.
> >
> > Thanks
> >
> >
> >
> > On Tue, Oct 29, 2013 at 7:54 PM, sebb <sebbaz@gmail.com> wrote:
> >
> >> In which case, maybe consider the related words:
> >>
> >> Apache Warp
> >> Apache Weft
> >>
> >> Just a thought.
> >>
> >> On 29 October 2013 22:14, Upayavira <uv@odoko.co.uk> wrote:
> >> > And Apache Wave too (which is what I first saw before I read the title
> >> > more carefully).
> >> >
> >> > Upayavira
> >> >
> >> > On Tue, Oct 29, 2013, at 09:12 PM, Matt Benson wrote:
> >> >> Hi,
> >> >>   I am concerned about potential confusion with Apache Commons Weaver
> >> >>   [1].
> >> >>
> >> >> Matt
> >> >>
> >> >> [1] https://commons.apache.org/proper/commons-weaver/
> >> >>
> >> >>
> >> >> On Tue, Oct 29, 2013 at 2:53 PM, Andreas Neumann <anew@apache.org>
> >> wrote:
> >> >>
> >> >> > I would like to propose Weave, an abstraction over Apache Hadoop®
> >> YARN to
> >> >> > reduce the complexity of developing distributed applications,
as an
> >> >> > Apache Incubator
> >> >> > podling.
> >> >> >
> >> >> > The proposal is included in plain text. I would also like to put
> >> this on
> >> >> > the wiki, but I appear to lack privileges to create pages. What
do
> I
> >> need
> >> >> > to do to get permission?
> >> >> >
> >> >> > -Andreas.
> >> >> >
> >> >> > Abstract
> >> >> > ========
> >> >> >
> >> >> > Weave is an abstraction over Apache Hadoop® YARN that reduces
the
> >> >> > complexity of developing distributed applications, allowing
> >> developers to
> >> >> > focus more on their business logic.
> >> >> >
> >> >> > Proposal
> >> >> > ========
> >> >> >
> >> >> > Weave is a set of libraries that reduces the complexity of
> developing
> >> >> > distributed applications. It exposes the distributed capabilities
> of
> >> Apache
> >> >> > Hadoop® YARN via a simple and intuitive programming model similar
> to
> >> Java
> >> >> > threads. Weave also has built-in capabilities required by many
> >> distributed
> >> >> > applications, such as real-time application logs and metrics
> >> collection,
> >> >> > application lifecycle management, and network service discovery.
> >> >> >
> >> >> > Background
> >> >> > ==========
> >> >> >
> >> >> > Hadoop YARN is a generic cluster resource manager that supports
any
> >> type of
> >> >> > distributed application. However, YARN’s interfaces are too
low
> >> level for
> >> >> > rapid application development. It requires a great deal of
> >> boilerplate code
> >> >> > even for a simple application, creating a high ramp up cost that
> can
> >> turn
> >> >> > developers away.
> >> >> >
> >> >> > Weave is designed to improve this situation with a programming
> model
> >> that
> >> >> > makes running distributed applications as easy as running Java
> >> threads.
> >> >> > With the abstraction provided by Weave, applications can be
> executed
> >> in
> >> >> > process threads during development and unit testing and then be
> >> deployed to
> >> >> > a YARN cluster without any modifications.
> >> >> >
> >> >> > Weave also has built-in support for real-time application logs
and
> >> metrics
> >> >> > collection, delegation token renewal, application lifecycle
> >> management, and
> >> >> > network service discovery. This greatly reduces the pain that
> >> developers
> >> >> > face when developing, debugging, deploying and monitoring
> distributed
> >> >> > applications.
> >> >> >
> >> >> > Weave is not a replacement for YARN, it’s a framework that operates
> >> on top
> >> >> > of YARN.
> >> >> >
> >> >> > Rationale
> >> >> > =========
> >> >> >
> >> >> > Developers who write YARN applications typically find themselves
> >> >> > implementing the same (or similar) boilerplate code over and over
> >> again for
> >> >> > every application. It makes sense to distill this common code
into
> a
> >> >> > reusable set of libraries that is perpetually maintained and
> >> improved by a
> >> >> > diverse community of developers.
> >> >> >
> >> >> > Weave’s simple thread-like programming model will enable many
Java
> >> >> > programmers to develop distributed applications. We believe that
> this
> >> >> > simplicity will attract developers who would otherwise be
> >> discouraged by
> >> >> > complexity, and many new use cases will emerge for the usage of
> YARN.
> >> >> >
> >> >> > Incubating Weave as an Apache project makes sense because Weave
is
> a
> >> >> > framework built on top of YARN, and Weave uses Apache Zookeeper,
> >> HDFS,
> >> >> > Kafka, and other Apache software (see the External Dependencies
> >> section).
> >> >> >
> >> >> > Current Status
> >> >> > ==============
> >> >> >
> >> >> > Weave was initially developed at Continuuity. The Weave codebase
is
> >> >> > currently hosted in a public repository at github.com, which will
> >> seed the
> >> >> > Apache git repository.
> >> >> >
> >> >> > Meritocracy
> >> >> > -----------
> >> >> > Our intent with this incubator proposal is to start building a
> >> diverse
> >> >> > developer community around Weave following the Apache meritocracy
> >> model.
> >> >> > Since Weave was initially developed in early 2013, we have had
fast
> >> >> > adoption and contributions within Continuuity. We are looking
> >> forward to
> >> >> > new contributors. We wish to build a community based on Apache's
> >> >> > meritocracy principles, working with those who contribute
> >> significantly to
> >> >> > the project and welcoming them to be committers both during the
> >> incubation
> >> >> > process and beyond.
> >> >> >
> >> >> > Community
> >> >> > ---------
> >> >> > Weave is currently being used internally at Continuuity and is
at
> >> the core
> >> >> > of our products. We hope to extend our contributor base
> >> significantly and
> >> >> > we will invite all who are interested in simplifying the
> development
> >> of
> >> >> > distributed applications to participate.
> >> >> >
> >> >> > Core Developers
> >> >> > ---------------
> >> >> > Weave is currently being developed by five engineers at
> Continuuity:
> >> >> >  Terence Yim, Andreas Neumann, Gary Helmling, Poorna Chandra and
> >> Albert
> >> >> > Shau.
> >> >> > Terence Yim is an Apache committer for Helix, Andreas is an Apache
> >> >> > committer and PMC member for Oozie, and Gary Helmling is an Apache
> >> >> > committer and PMC member for HBase. Poorna Chandra and Albert
Shau
> >> have
> >> >> > made many contributions to Weave.
> >> >> >
> >> >> > Alignment
> >> >> > ---------
> >> >> > The ASF is the natural choice to host the Weave project as its
goal
> >> of
> >> >> > encouraging community-driven open source projects fits with our
> >> vision for
> >> >> > Weave.
> >> >> >
> >> >> > Additionally, many other projects with which we are familiar and
> >> expect
> >> >> > Weave to integrate with, such as ZooKeeper, YARN, HDFS, log4j,
and
> >> others
> >> >> > mentioned in the External Dependencies section are Apache projects,
> >> and
> >> >> > Weave will benefit by close proximity to them.
> >> >> >
> >> >> > Known Risks
> >> >> > ===========
> >> >> >
> >> >> > Orphaned Products
> >> >> > -----------------
> >> >> > There is very little risk of Weave being orphaned, as it is a
key
> >> part of
> >> >> > Continuuity’s products. The core Weave developers plan to continue
> >> to work
> >> >> > on Weave, and Continuuity has funding in place to support their
> >> efforts
> >> >> > going forward.
> >> >> > Many other Big Data companies can benefit from Weave, and we have
> >> already
> >> >> > received interest from various entities that would like to use
and
> >> >> > contribute to Weave.
> >> >> >
> >> >> > Inexperience with Open Source
> >> >> > -----------------------------
> >> >> > Several of the core developers have experience with open source
> >> >> > development. Terence Yim, Andreas Neumann and Gary Helmling are
> >> currently
> >> >> > Apache committers for Helix, Oozie and HBase respectively
> >> >> > Homogeneous Developers
> >> >> >
> >> >> > The current core developers are all Continuuity employees. However,
> >> we
> >> >> > intend to establish a developer community that includes independent
> >> and
> >> >> > corporate contributors. We are encouraging new contributors via
our
> >> mailing
> >> >> > lists, public presentations, and personal contacts, and we will
> >> continue to
> >> >> > do so. Various entities have already expressed interest in becoming
> >> >> > involved with Weave.
> >> >> >
> >> >> > Reliance on Salaried Developers
> >> >> > -------------------------------
> >> >> > Currently, these developers are paid to work on Weave. Once the
> >> project has
> >> >> > built a community, we expect to attract committers, developers
and
> >> >> > community other the current core developers. However, because
> >> Continuuity
> >> >> > products use Weave internally, the reliance on salaried developers
> is
> >> >> > unlikely to change, at least in the near term.
> >> >> >
> >> >> > Relationships with Other Apache Products
> >> >> > ----------------------------------------
> >> >> > Weave is deeply integrated with Apache projects. Weave uses Apache
> >> YARN as
> >> >> > its underlying resource management and task scheduling system
and
> >> Apache
> >> >> > Zookeeper for coordination. In addition, Weave uses Apache HDFS
and
> >> Apache
> >> >> > Kafka. A number of other Apache projects are Weave dependencies
and
> >> are
> >> >> > listed in the External Dependencies section.
> >> >> >
> >> >> > An Excessive Fascination with the Apache Brand
> >> >> > ----------------------------------------------
> >> >> > While we respect the reputation of the Apache brand and have no
> >> doubt that
> >> >> > it will attract contributors and users, our interest is primarily
> to
> >> give
> >> >> > Weave a solid home as an open source project following an
> established
> >> >> > development model. We have also given additional reasons in the
> >> Rationale
> >> >> > and Alignment sections.
> >> >> >
> >> >> > Documentation
> >> >> > =============
> >> >> >
> >> >> > https://github.com/continuuity/weave
> >> >> >
> >> >> > Initial Source
> >> >> > ==============
> >> >> >
> >> >> > The Weave codebase is currently hosted at
> >> >> > https://github.com/continuuity/weave.
> >> >> >
> >> >> > External Dependencies
> >> >> > =====================
> >> >> >
> >> >> > The dependencies all have Apache-compatible licenses:
> >> >> > • avro (Apache 2.0)
> >> >> > • hadoop (Apache 2.0)
> >> >> > • gson (Apache 2.0)
> >> >> > • guava-libraries (Apache 2.0)
> >> >> > • hbase (Apache 2.0)
> >> >> > • hdfs (Apache 2.0)
> >> >> > • kafka (Apache 2.0)
> >> >> > • netty (Apache 2.0)
> >> >> > • snappy-java (Apache 2.0)
> >> >> > • yarn (Apache 2.0)
> >> >> > • zookeeper (Apache 2.0)
> >> >> > • asm (BSD)
> >> >> > • junit (EPL v1.0)
> >> >> > • logback (EPL v1.0 )
> >> >> > • slf4j (MIT)
> >> >> >
> >> >> > Cryptography
> >> >> > ============
> >> >> >
> >> >> > Weave will depend on secure Hadoop, which can optionally use
> >> Kerberos.
> >> >> >
> >> >> > Required Resources
> >> >> > ==================
> >> >> >
> >> >> > Mailing Lists
> >> >> > -------------
> >> >> > - weave-private for private PMC discussions (with moderated
> >> subscriptions)
> >> >> > - weave-dev for technical discussions among contributors
> >> >> > - weave-commits for notification about commits
> >> >> >
> >> >> > Subversion Directory
> >> >> > --------------------
> >> >> > Git is the preferred source control system: git://
> >> git.apache.org/weave
> >> >> >
> >> >> > Issue Tracking
> >> >> > --------------
> >> >> > JIRA Weave (WEAVE)
> >> >> >
> >> >> > Other Resources
> >> >> > ---------------
> >> >> > The existing code already has unit tests, so we would like a Hudson
> >> >> > instance to run them whenever a new patch is submitted. This can
be
> >> added
> >> >> > after project creation.
> >> >> >
> >> >> > Initial Committers
> >> >> > ==================
> >> >> >
> >> >> > • Terence Yim
> >> >> > • Andreas Neumann
> >> >> > • Gary Helmling
> >> >> > • Poorna Chandra
> >> >> > • Albert Shau
> >> >> >
> >> >> > Affiliations
> >> >> > ============
> >> >> >
> >> >> > • Terence Yim (Continuuity)
> >> >> > • Andreas Neumann (Continuuity)
> >> >> > • Gary Helmling (Continuuity)
> >> >> > • Poorna Chandra (Continuuity)
> >> >> > • Albert Shau (Continuuity)
> >> >> >
> >> >> > Sponsors
> >> >> > ========
> >> >> >
> >> >> > Champion
> >> >> > --------
> >> >> > Vinod K <vinodkv at apache dot org> (Apache Member)
> >> >> >
> >> >> > Nominated Mentors
> >> >> > -----------------
> >> >> > • Arun C Murthy <acmurthy at apache dot org>
> >> >> > • Tom White <tomwhite at apache dot org>
> >> >> > • Patrick Hunt <phunt at apache dot org>
> >> >> >
> >> >> > Sponsoring Entity
> >> >> > -----------------
> >> >> > We are requesting that the Incubator sponsor this project.
> >> >> >
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> > For additional commands, e-mail: general-help@incubator.apache.org
> >> >
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message