incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Rowe <sar...@gmail.com>
Subject Re: [PROPOSAL] Weave for Apache Incubator
Date Wed, 30 Oct 2013 18:16:59 GMT
Random lurker project name suggestion: yarnbomb - see <
http://en.m.wikipedia.org/wiki/Yarn_bombing>

/me relurking

Steve
On Oct 30, 2013 2:00 PM, "Donald Whytock" <dwhytock@apache.org> wrote:

> Knit, crochet, macrame?
>
> Though, honestly, pulling lower-level components together into higher-level
> ones sounds a little like granny squares...
>
> Don
>
>
> On Wed, Oct 30, 2013 at 1:53 AM, Andreas Neumann <anew@apache.org> wrote:
>
> > Pardon my ignorance, I just looked it up and realized that warp and weft
> > are indeed related to weaving, so they might work.
> > I do have the impression, though, that most people would associate Warp
> > with the speed of light and not with weaving.
> >
> > -Andreas.
> >
> > On Wed, Oct 30, 2013 at 1:41 AM, Andreas Neumann <anew@apache.org>
> wrote:
> >
> > > Thanks for pointing out these similarities; we were not aware of
> Commons
> > > Weaver. Given that Weaver is a sub-project of Commons, would the
> > similarity
> > > be tolerable? Also, since Weave and Wave are pronounced quite
> > differently,
> > > I am hoping that they are perceived as different enough.
> > >
> > > The name Weave is motivated from the name of YARN - it takes the
> > > complexity out of yarn by "weaving" it into a simple pattern. Warp and
> > Weft
> > > don't really convey this meaning.
> > >
> > > If the concern about the name similarity is really strong, we will try
> to
> > > find another name, and we are open to suggestions. But please do also
> > > consider the motivation for this naming.
> > >
> > > Thanks
> > >
> > >
> > >
> > > On Tue, Oct 29, 2013 at 7:54 PM, sebb <sebbaz@gmail.com> wrote:
> > >
> > >> In which case, maybe consider the related words:
> > >>
> > >> Apache Warp
> > >> Apache Weft
> > >>
> > >> Just a thought.
> > >>
> > >> On 29 October 2013 22:14, Upayavira <uv@odoko.co.uk> wrote:
> > >> > And Apache Wave too (which is what I first saw before I read the
> title
> > >> > more carefully).
> > >> >
> > >> > Upayavira
> > >> >
> > >> > On Tue, Oct 29, 2013, at 09:12 PM, Matt Benson wrote:
> > >> >> Hi,
> > >> >>   I am concerned about potential confusion with Apache Commons
> Weaver
> > >> >>   [1].
> > >> >>
> > >> >> Matt
> > >> >>
> > >> >> [1] https://commons.apache.org/proper/commons-weaver/
> > >> >>
> > >> >>
> > >> >> On Tue, Oct 29, 2013 at 2:53 PM, Andreas Neumann <anew@apache.org>
> > >> wrote:
> > >> >>
> > >> >> > I would like to propose Weave, an abstraction over Apache
Hadoop®
> > >> YARN to
> > >> >> > reduce the complexity of developing distributed applications,
as
> an
> > >> >> > Apache Incubator
> > >> >> > podling.
> > >> >> >
> > >> >> > The proposal is included in plain text. I would also like
to put
> > >> this on
> > >> >> > the wiki, but I appear to lack privileges to create pages.
What
> do
> > I
> > >> need
> > >> >> > to do to get permission?
> > >> >> >
> > >> >> > -Andreas.
> > >> >> >
> > >> >> > Abstract
> > >> >> > ========
> > >> >> >
> > >> >> > Weave is an abstraction over Apache Hadoop® YARN that reduces
the
> > >> >> > complexity of developing distributed applications, allowing
> > >> developers to
> > >> >> > focus more on their business logic.
> > >> >> >
> > >> >> > Proposal
> > >> >> > ========
> > >> >> >
> > >> >> > Weave is a set of libraries that reduces the complexity of
> > developing
> > >> >> > distributed applications. It exposes the distributed capabilities
> > of
> > >> Apache
> > >> >> > Hadoop® YARN via a simple and intuitive programming model
similar
> > to
> > >> Java
> > >> >> > threads. Weave also has built-in capabilities required by
many
> > >> distributed
> > >> >> > applications, such as real-time application logs and metrics
> > >> collection,
> > >> >> > application lifecycle management, and network service discovery.
> > >> >> >
> > >> >> > Background
> > >> >> > ==========
> > >> >> >
> > >> >> > Hadoop YARN is a generic cluster resource manager that supports
> any
> > >> type of
> > >> >> > distributed application. However, YARN’s interfaces are
too low
> > >> level for
> > >> >> > rapid application development. It requires a great deal of
> > >> boilerplate code
> > >> >> > even for a simple application, creating a high ramp up cost
that
> > can
> > >> turn
> > >> >> > developers away.
> > >> >> >
> > >> >> > Weave is designed to improve this situation with a programming
> > model
> > >> that
> > >> >> > makes running distributed applications as easy as running
Java
> > >> threads.
> > >> >> > With the abstraction provided by Weave, applications can
be
> > executed
> > >> in
> > >> >> > process threads during development and unit testing and then
be
> > >> deployed to
> > >> >> > a YARN cluster without any modifications.
> > >> >> >
> > >> >> > Weave also has built-in support for real-time application
logs
> and
> > >> metrics
> > >> >> > collection, delegation token renewal, application lifecycle
> > >> management, and
> > >> >> > network service discovery. This greatly reduces the pain
that
> > >> developers
> > >> >> > face when developing, debugging, deploying and monitoring
> > distributed
> > >> >> > applications.
> > >> >> >
> > >> >> > Weave is not a replacement for YARN, it’s a framework that
> operates
> > >> on top
> > >> >> > of YARN.
> > >> >> >
> > >> >> > Rationale
> > >> >> > =========
> > >> >> >
> > >> >> > Developers who write YARN applications typically find themselves
> > >> >> > implementing the same (or similar) boilerplate code over
and over
> > >> again for
> > >> >> > every application. It makes sense to distill this common
code
> into
> > a
> > >> >> > reusable set of libraries that is perpetually maintained
and
> > >> improved by a
> > >> >> > diverse community of developers.
> > >> >> >
> > >> >> > Weave’s simple thread-like programming model will enable
many
> Java
> > >> >> > programmers to develop distributed applications. We believe
that
> > this
> > >> >> > simplicity will attract developers who would otherwise be
> > >> discouraged by
> > >> >> > complexity, and many new use cases will emerge for the usage
of
> > YARN.
> > >> >> >
> > >> >> > Incubating Weave as an Apache project makes sense because
Weave
> is
> > a
> > >> >> > framework built on top of YARN, and Weave uses Apache Zookeeper,
> > >> HDFS,
> > >> >> > Kafka, and other Apache software (see the External Dependencies
> > >> section).
> > >> >> >
> > >> >> > Current Status
> > >> >> > ==============
> > >> >> >
> > >> >> > Weave was initially developed at Continuuity. The Weave codebase
> is
> > >> >> > currently hosted in a public repository at github.com, which
> will
> > >> seed the
> > >> >> > Apache git repository.
> > >> >> >
> > >> >> > Meritocracy
> > >> >> > -----------
> > >> >> > Our intent with this incubator proposal is to start building
a
> > >> diverse
> > >> >> > developer community around Weave following the Apache meritocracy
> > >> model.
> > >> >> > Since Weave was initially developed in early 2013, we have
had
> fast
> > >> >> > adoption and contributions within Continuuity. We are looking
> > >> forward to
> > >> >> > new contributors. We wish to build a community based on Apache's
> > >> >> > meritocracy principles, working with those who contribute
> > >> significantly to
> > >> >> > the project and welcoming them to be committers both during
the
> > >> incubation
> > >> >> > process and beyond.
> > >> >> >
> > >> >> > Community
> > >> >> > ---------
> > >> >> > Weave is currently being used internally at Continuuity and
is at
> > >> the core
> > >> >> > of our products. We hope to extend our contributor base
> > >> significantly and
> > >> >> > we will invite all who are interested in simplifying the
> > development
> > >> of
> > >> >> > distributed applications to participate.
> > >> >> >
> > >> >> > Core Developers
> > >> >> > ---------------
> > >> >> > Weave is currently being developed by five engineers at
> > Continuuity:
> > >> >> >  Terence Yim, Andreas Neumann, Gary Helmling, Poorna Chandra
and
> > >> Albert
> > >> >> > Shau.
> > >> >> > Terence Yim is an Apache committer for Helix, Andreas is
an
> Apache
> > >> >> > committer and PMC member for Oozie, and Gary Helmling is
an
> Apache
> > >> >> > committer and PMC member for HBase. Poorna Chandra and Albert
> Shau
> > >> have
> > >> >> > made many contributions to Weave.
> > >> >> >
> > >> >> > Alignment
> > >> >> > ---------
> > >> >> > The ASF is the natural choice to host the Weave project as
its
> goal
> > >> of
> > >> >> > encouraging community-driven open source projects fits with
our
> > >> vision for
> > >> >> > Weave.
> > >> >> >
> > >> >> > Additionally, many other projects with which we are familiar
and
> > >> expect
> > >> >> > Weave to integrate with, such as ZooKeeper, YARN, HDFS, log4j,
> and
> > >> others
> > >> >> > mentioned in the External Dependencies section are Apache
> projects,
> > >> and
> > >> >> > Weave will benefit by close proximity to them.
> > >> >> >
> > >> >> > Known Risks
> > >> >> > ===========
> > >> >> >
> > >> >> > Orphaned Products
> > >> >> > -----------------
> > >> >> > There is very little risk of Weave being orphaned, as it
is a key
> > >> part of
> > >> >> > Continuuity’s products. The core Weave developers plan
to
> continue
> > >> to work
> > >> >> > on Weave, and Continuuity has funding in place to support
their
> > >> efforts
> > >> >> > going forward.
> > >> >> > Many other Big Data companies can benefit from Weave, and
we have
> > >> already
> > >> >> > received interest from various entities that would like to
use
> and
> > >> >> > contribute to Weave.
> > >> >> >
> > >> >> > Inexperience with Open Source
> > >> >> > -----------------------------
> > >> >> > Several of the core developers have experience with open
source
> > >> >> > development. Terence Yim, Andreas Neumann and Gary Helmling
are
> > >> currently
> > >> >> > Apache committers for Helix, Oozie and HBase respectively
> > >> >> > Homogeneous Developers
> > >> >> >
> > >> >> > The current core developers are all Continuuity employees.
> However,
> > >> we
> > >> >> > intend to establish a developer community that includes
> independent
> > >> and
> > >> >> > corporate contributors. We are encouraging new contributors
via
> our
> > >> mailing
> > >> >> > lists, public presentations, and personal contacts, and we
will
> > >> continue to
> > >> >> > do so. Various entities have already expressed interest in
> becoming
> > >> >> > involved with Weave.
> > >> >> >
> > >> >> > Reliance on Salaried Developers
> > >> >> > -------------------------------
> > >> >> > Currently, these developers are paid to work on Weave. Once
the
> > >> project has
> > >> >> > built a community, we expect to attract committers, developers
> and
> > >> >> > community other the current core developers. However, because
> > >> Continuuity
> > >> >> > products use Weave internally, the reliance on salaried
> developers
> > is
> > >> >> > unlikely to change, at least in the near term.
> > >> >> >
> > >> >> > Relationships with Other Apache Products
> > >> >> > ----------------------------------------
> > >> >> > Weave is deeply integrated with Apache projects. Weave uses
> Apache
> > >> YARN as
> > >> >> > its underlying resource management and task scheduling system
and
> > >> Apache
> > >> >> > Zookeeper for coordination. In addition, Weave uses Apache
HDFS
> and
> > >> Apache
> > >> >> > Kafka. A number of other Apache projects are Weave dependencies
> and
> > >> are
> > >> >> > listed in the External Dependencies section.
> > >> >> >
> > >> >> > An Excessive Fascination with the Apache Brand
> > >> >> > ----------------------------------------------
> > >> >> > While we respect the reputation of the Apache brand and have
no
> > >> doubt that
> > >> >> > it will attract contributors and users, our interest is primarily
> > to
> > >> give
> > >> >> > Weave a solid home as an open source project following an
> > established
> > >> >> > development model. We have also given additional reasons
in the
> > >> Rationale
> > >> >> > and Alignment sections.
> > >> >> >
> > >> >> > Documentation
> > >> >> > =============
> > >> >> >
> > >> >> > https://github.com/continuuity/weave
> > >> >> >
> > >> >> > Initial Source
> > >> >> > ==============
> > >> >> >
> > >> >> > The Weave codebase is currently hosted at
> > >> >> > https://github.com/continuuity/weave.
> > >> >> >
> > >> >> > External Dependencies
> > >> >> > =====================
> > >> >> >
> > >> >> > The dependencies all have Apache-compatible licenses:
> > >> >> > • avro (Apache 2.0)
> > >> >> > • hadoop (Apache 2.0)
> > >> >> > • gson (Apache 2.0)
> > >> >> > • guava-libraries (Apache 2.0)
> > >> >> > • hbase (Apache 2.0)
> > >> >> > • hdfs (Apache 2.0)
> > >> >> > • kafka (Apache 2.0)
> > >> >> > • netty (Apache 2.0)
> > >> >> > • snappy-java (Apache 2.0)
> > >> >> > • yarn (Apache 2.0)
> > >> >> > • zookeeper (Apache 2.0)
> > >> >> > • asm (BSD)
> > >> >> > • junit (EPL v1.0)
> > >> >> > • logback (EPL v1.0 )
> > >> >> > • slf4j (MIT)
> > >> >> >
> > >> >> > Cryptography
> > >> >> > ============
> > >> >> >
> > >> >> > Weave will depend on secure Hadoop, which can optionally
use
> > >> Kerberos.
> > >> >> >
> > >> >> > Required Resources
> > >> >> > ==================
> > >> >> >
> > >> >> > Mailing Lists
> > >> >> > -------------
> > >> >> > - weave-private for private PMC discussions (with moderated
> > >> subscriptions)
> > >> >> > - weave-dev for technical discussions among contributors
> > >> >> > - weave-commits for notification about commits
> > >> >> >
> > >> >> > Subversion Directory
> > >> >> > --------------------
> > >> >> > Git is the preferred source control system: git://
> > >> git.apache.org/weave
> > >> >> >
> > >> >> > Issue Tracking
> > >> >> > --------------
> > >> >> > JIRA Weave (WEAVE)
> > >> >> >
> > >> >> > Other Resources
> > >> >> > ---------------
> > >> >> > The existing code already has unit tests, so we would like
a
> Hudson
> > >> >> > instance to run them whenever a new patch is submitted. This
can
> be
> > >> added
> > >> >> > after project creation.
> > >> >> >
> > >> >> > Initial Committers
> > >> >> > ==================
> > >> >> >
> > >> >> > • Terence Yim
> > >> >> > • Andreas Neumann
> > >> >> > • Gary Helmling
> > >> >> > • Poorna Chandra
> > >> >> > • Albert Shau
> > >> >> >
> > >> >> > Affiliations
> > >> >> > ============
> > >> >> >
> > >> >> > • Terence Yim (Continuuity)
> > >> >> > • Andreas Neumann (Continuuity)
> > >> >> > • Gary Helmling (Continuuity)
> > >> >> > • Poorna Chandra (Continuuity)
> > >> >> > • Albert Shau (Continuuity)
> > >> >> >
> > >> >> > Sponsors
> > >> >> > ========
> > >> >> >
> > >> >> > Champion
> > >> >> > --------
> > >> >> > Vinod K <vinodkv at apache dot org> (Apache Member)
> > >> >> >
> > >> >> > Nominated Mentors
> > >> >> > -----------------
> > >> >> > • Arun C Murthy <acmurthy at apache dot org>
> > >> >> > • Tom White <tomwhite at apache dot org>
> > >> >> > • Patrick Hunt <phunt at apache dot org>
> > >> >> >
> > >> >> > Sponsoring Entity
> > >> >> > -----------------
> > >> >> > We are requesting that the Incubator sponsor this project.
> > >> >> >
> > >> >
> > >> >
> ---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >> > For additional commands, e-mail: general-help@incubator.apache.org
> > >> >
> > >>
> > >> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > >> For additional commands, e-mail: general-help@incubator.apache.org
> > >>
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message