incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: [PROPOSAL] Curator for the Apache Incubator
Date Tue, 26 Feb 2013 19:39:03 GMT
On Tue, Feb 26, 2013 at 10:30 AM, Mattmann, Chris A (388J)
<chris.a.mattmann@jpl.nasa.gov> wrote:
> Hi Guys,
>
> Sorry I have to ask the question here. If the mentors consist of PMC
> members on ZK
> (at least Pat and Mahadev), what's the problem with creating a branch in
> ZK and just
> having the code be there and getting the Curator proposed committers as
> committers in
> ZK ville, and hopefully PMC members soon thereafter.
>
> I have to agree with Greg here. Seems like Incubation is something that
> might not be
> needed. If the end result of this after N months is that Curator
> "graduates" into another
> set of flipping bylaw updates, and more legislation that makes these
> people unofficial
> PMC members on ZK, then I'm double triple -1 with 50 piles of coal on top.
> Yes, I'm
> still remembering the HCatalog/Hive thing here - I still don't get that.
>
> Either: (a) define Curator to be its own separate project/community, with
> a goal of TLP;
> or (b) nix Incubation and just make these guys part of ZK, on a branch
> now.

Hi Chris. Unfortunately it's not clear cut to me where Curator should
go, perhaps because I'm close to it. For example the "Curator as TLP"
argument might sound like this:

1) curator has been around for a while, people seem to like it
2) it shares no code with ZK today, in the sense that it's a consumer
of the ZK APIs, what we call "recipes". No code duplication. Granted
it does try to make writing clients easier.
3) (afaik) none of the current ZK committers have contributed to
Curator, when this proposal came up previously for discussion on the
ZK dev list there wasn't strong interest to pull it into ZK itself
4) the initial committers on curator don't have experience at Apache

On the contrary I personally think it might make sense to have a
combined community with Curator and ZooKeeper together, but I'm not
sure.

My hope was that Curator could come into the incubator, work through
the IP issues and other incubation activities, learn about Apache
process, and then decide based on the new found insight/experience.

The proposal guide talks about this and makes it clear (at least to
me) that leaving graduation open is not only an option, but a
requirement "Note that the final destination within the Apache
organizational structure will be decided upon graduation." Given this
I don't see why we'd artificially constrain things up front as you've
suggested.

Patrick


>
> On 2/26/13 9:40 AM, "Benson Margulies" <bimargulies@gmail.com> wrote:
>
>>On Tue, Feb 26, 2013 at 12:22 PM, Patrick Hunt <phunt@apache.org> wrote:
>>> On Tue, Feb 26, 2013 at 8:32 AM, Benson Margulies
>>><bimargulies@gmail.com> wrote:
>>>> If you think that the right destination for curator is as part of ZK,
>>>> then it would be good to see substantive participation of the ZK PMC
>>>> in the incubation. The goal should be to 'graduate' by having the
>>>> curator community be granted karma at ZK and the code folded in. This
>>>> would require, I think, the ZK community to supply at least one
>>>> mentor, and to have a plan in advance for the eventual votes based on
>>>> success in the incubator.
>>>
>>> Hi Benson. What you're suggesting matches my current thinking.
>>>
>>> The three Curator mentors are as follows: Mahadev and I (champion) are
>>> both PMC members on ZK, Enis Söztutar is active on HBase and
>>> interested in using Curator for that project (which already uses ZK
>>> heavily).
>>
>>OK, that's lovely.
>>
>>
>>>
>>> Patrick
>>>
>>>> On Tue, Feb 26, 2013 at 10:53 AM, Henry Saputra
>>>><henry.saputra@gmail.com> wrote:
>>>>> Ah no, I was not suggesting about Curator to become subproject of ZK.
>>>>>I
>>>>> just afraid that if Curator is going as incubator it will end up as
>>>>>sub of
>>>>> ZK as merging process.
>>>>>
>>>>> Like Greg has mentioned in another reply, I would prefer Curator to be
>>>>> merged as a higher level ZK client. Surely project like HBase and
>>>>>others
>>>>> that relying on ZK would appreciate simpler client to ZK.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Henry
>>>>>
>>>>>
>>>>> On Mon, Feb 25, 2013 at 11:23 PM, Patrick Hunt <phunt@apache.org>
>>>>>wrote:
>>>>>
>>>>>> On Mon, Feb 25, 2013 at 11:01 PM, Henry Saputra
>>>>>><henry.saputra@gmail.com>
>>>>>> wrote:
>>>>>> > So isnt this similar to HCatalog which relying on Hive metadata
>>>>>>service
>>>>>> > that ends up as sub project of Apache Hive?
>>>>>> >
>>>>>>
>>>>>> I was against having Curator as a sub when it came up on the original
>>>>>> discussion thread, I still am.
>>>>>>
>>>>>> Patrick
>>>>>>
>>>>>> >
>>>>>> >
>>>>>> > On Mon, Feb 25, 2013 at 7:10 PM, Greg Stein <gstein@gmail.com>
>>>>>>wrote:
>>>>>> >
>>>>>> >> My concern is that we're looking at two "new" committers,
rather
>>>>>>than
>>>>>> >> a Curator community. Following normal Incubator work, Curator
>>>>>>would
>>>>>> >> build a community for itself. But then we'd have a community
>>>>>> >> *distinct* from that of Zookeeper. And it really looks like
this
>>>>>> >> should be part of Zookeeper itself -- a more capable and
>>>>>>easier-to-use
>>>>>> >> client.
>>>>>> >>
>>>>>> >> So I question the incubation of this. Why do we want to
build a
>>>>>> >> new/separate project? Why isn't this just part of Zookeeper
right
>>>>>>from
>>>>>> >> the start?
>>>>>> >>
>>>>>> >> I would suggest that this work is placed on a branch within
>>>>>>Zookeeper.
>>>>>> >> That Jordan and Jay become committers on that branch (not
>>>>>>necessarily
>>>>>> >> Zookeeper trunk). Over time, the branch can be folded into
trunk,
>>>>>> >> along with all the various tests, doc, and other artifacts
that I
>>>>>>see
>>>>>> >> in the GitHub repository. And hopefully that Jordan and
Jay become
>>>>>> >> regular committers (and PMC members!) of the Zookeeper project
>>>>>>itself.
>>>>>> >>
>>>>>> >> The current Zookeeper client can remain for backwards compat,
and
>>>>>>the
>>>>>> >> Curator work can become the next-gen client.
>>>>>> >>
>>>>>> >> Honestly, in my opnion, incubating this project seems like
it
>>>>>>would
>>>>>> >> create a distinct community, and really doesn't seem like
it would
>>>>>> >> serve the Zookeeper community.
>>>>>> >>
>>>>>> >> All that said, I am not familiar with the Zookeeper or Curator
>>>>>> >> communities. But from this read, I don't think Incubation
is the
>>>>>>right
>>>>>> >> approach. I would rather push for a more direct incorporation
of
>>>>>> >> Curator directly into Zookeeper. (use the short-form IP
>>>>>>clearance) ...
>>>>>> >> so, unless somebody can help me understand the communities
and
>>>>>> >> situation better, I'm -1 (binding) on this incubation. I'd
rather
>>>>>>see
>>>>>> >> combined, rather than distinct, communities from the start.
>>>>>> >>
>>>>>> >> Thanks,
>>>>>> >> -g
>>>>>> >>
>>>>>> >> On Mon, Feb 25, 2013 at 8:14 PM, Jordan Zimmerman
>>>>>> >> <jordan@jordanzimmerman.com> wrote:
>>>>>> >> > Hello,
>>>>>> >> >
>>>>>> >> > I would like to propose that Curator to be an Apache
Incubator
>>>>>> project.
>>>>>> >> >
>>>>>> >> > The proposal can be found here:
>>>>>> >> http://wiki.apache.org/incubator/CuratorProposal
>>>>>> >> >
>>>>>> >> > I have included the contents of the proposal below.
>>>>>> >> >
>>>>>> >> > Sincerely,
>>>>>> >> >
>>>>>> >> > Jordan Zimmerman
>>>>>> >> >
>>>>>> >> > ===================
>>>>>> >> >
>>>>>> >> > = Curator - ZooKeeper client wrapper and rich ZooKeeper
>>>>>>framework =
>>>>>> >> >
>>>>>> >> > == Abstract ==
>>>>>> >> >
>>>>>> >> > Curator is a set of Java libraries that make using
Apache
>>>>>>ZooKeeper
>>>>>> much
>>>>>> >> easier. While ZooKeeper comes bundled with a Java client,
using
>>>>>>the
>>>>>> client
>>>>>> >> is non-trivial and error prone.
>>>>>> >> >
>>>>>> >> > == Proposal ==
>>>>>> >> >
>>>>>> >> > Curator is a set of Java libraries that make using
Apache
>>>>>>ZooKeeper
>>>>>> much
>>>>>> >> easier. While ZooKeeper comes bundled with a Java client,
using
>>>>>>the
>>>>>> client
>>>>>> >> is non-trivial and error prone. It consists of three components
>>>>>>that
>>>>>> build
>>>>>> >> on each other. Curator Client is a replacement for the bundled
>>>>>>ZooKeeper
>>>>>> >> class that takes care of some low-level housekeeping and
provides
>>>>>>some
>>>>>> >> useful utilities. Curator Framework is a high-level API
that
>>>>>>greatly
>>>>>> >> simplifies using ZooKeeper. It adds many features that build
on
>>>>>> ZooKeeper
>>>>>> >> and handles the complexity of managing connections to the
>>>>>>ZooKeeper
>>>>>> cluster
>>>>>> >> and retrying operations. Curator Recipes consists of
>>>>>>implementations of
>>>>>> >> some of the common ZooKeeper ³recipes². Additionally,
Curator
>>>>>>Test is
>>>>>> >> included which includes utilities to help with unit testing
>>>>>> ZooKeeper-based
>>>>>> >> applications.
>>>>>> >> >
>>>>>> >> > == Background ==
>>>>>> >> >
>>>>>> >> > Curator was initially developed by Netflix to make
writing
>>>>>> >> ZooKeeper-based applications easier and more reliable. Curator
was
>>>>>> >> open-sourced by Netflix on !GitHub as an Apache 2.0 licensed
>>>>>>project in
>>>>>> >> July 2011. During this time Curator has been formally released
>>>>>>many
>>>>>> times
>>>>>> >> and has gained widespread adoption.
>>>>>> >> >
>>>>>> >> > == Rationale ==
>>>>>> >> >
>>>>>> >> > New users of ZooKeeper are surprised to learn that
a significant
>>>>>> amount
>>>>>> >> of connection management must be done manually. For example,
when
>>>>>>the
>>>>>> >> ZooKeeper client connects to the ensemble it must negotiate
a new
>>>>>> session,
>>>>>> >> etc. This takes some time. If you use a ZooKeeper client
API
>>>>>>before the
>>>>>> >> connection process has completed, ZooKeeper will throw an
>>>>>>exception.
>>>>>> These
>>>>>> >> types of exceptions are referred to as ³recoverable² errors.
>>>>>> >> > Curator automatically handles connection management,
greatly
>>>>>> simplifying
>>>>>> >> client code. Instead of directly using the ZooKeeper APIs
you use
>>>>>> Curator
>>>>>> >> APIs that internally check for connection completion and
wrap each
>>>>>> >> ZooKeeper API in a retry loop. Curator uses a retry mechanism
to
>>>>>>handle
>>>>>> >> recoverable errors and automatically retry operations. The
method
>>>>>>of
>>>>>> retry
>>>>>> >> is customizable. Curator comes bundled with several
>>>>>>implementations
>>>>>> >> (ExponentialBackoffRetry, etc.) or custom implementations
can be
>>>>>> written.
>>>>>> >> >
>>>>>> >> > The ZooKeeper documentation describes many possible
uses for
>>>>>>ZooKeeper
>>>>>> >> calling each a ³recipe². While the distribution comes
bundled
>>>>>>with a few
>>>>>> >> implementations of these recipes, most ZooKeeper users will
need
>>>>>>to
>>>>>> >> manually implement one or more of the recipes. Implementing
a
>>>>>>ZooKeeper
>>>>>> >> recipe is not trivial. Besides the connection handling issues,
>>>>>>there are
>>>>>> >> numerous edge cases that are not well documented that must
be
>>>>>> considered.
>>>>>> >> For example, many recipes require that an ephemeral-sequential
>>>>>>node be
>>>>>> >> created. New users of ZooKeeper will not know that there
is an
>>>>>>edge
>>>>>> case in
>>>>>> >> ephemeral-sequential node creation that requires you to
put a
>>>>>>special
>>>>>> >> ³marker² in the node¹s name so that you can search for
the
>>>>>>created node
>>>>>> if
>>>>>> >> an I/O failure occurs. This is but one of many edge cases
that
>>>>>>are not
>>>>>> well
>>>>>> >> documented but are handled by Curator.
>>>>>> >> >
>>>>>> >> > = Current Status =
>>>>>> >> >
>>>>>> >> > == Meritocracy ==
>>>>>> >> >
>>>>>> >> > Curator was initially developed by Jordan Zimmerman
in 2011 at
>>>>>> Netflix.
>>>>>> >> Developers external to Netflix provided feedback, suggested
>>>>>>features and
>>>>>> >> fixes and implemented extensions of Curator. Netflix's
>>>>>>engineering team
>>>>>> has
>>>>>> >> since maintained the project and has been dedicated towards
its
>>>>>> >> improvement. Contributors to Curator include developers
from
>>>>>>multiple
>>>>>> >> organizations around the world. Curator will be a meritocracy
as
>>>>>>it
>>>>>> enters
>>>>>> >> the Incubator and beyond.
>>>>>> >> >
>>>>>> >> > == Community ==
>>>>>> >> >
>>>>>> >> > Curator is currently used by a number of organizations
all over
>>>>>>the
>>>>>> >> world. Curator has an active and growing user and developer
>>>>>>community
>>>>>> with
>>>>>> >> active participation in the [[
>>>>>> http://groups.google.com/group/curator-users]]
>>>>>> >> mailing list and at its !Github home: [[
>>>>>> >> https://github.com/Netflix/curator]].
>>>>>> >> >
>>>>>> >> > Since open sourcing the project, there have been fifteen
>>>>>>individuals
>>>>>> >> from various organizations who have contributed code.
>>>>>> >> >
>>>>>> >> > == Core Developers ==
>>>>>> >> >
>>>>>> >> > The core developers for Curator are:
>>>>>> >> >  * Jordan Zimmerman
>>>>>> >> >  * Jay Zarfoss
>>>>>> >> >
>>>>>> >> > Jordan has contributed towards Apache ZooKeeper and
both Jordan
>>>>>>and
>>>>>> Jay
>>>>>> >> are familiar with Apache principles and philosophy for community
>>>>>>driven
>>>>>> >> software development.
>>>>>> >> >
>>>>>> >> > == Alignment ==
>>>>>> >> >
>>>>>> >> > Curator is a natural complement for Apache ZooKeeper.
Java
>>>>>>users of
>>>>>> >> ZooKeeper will naturally want to use Curator. When Curator
>>>>>>graduates
>>>>>> from
>>>>>> >> Incubator it may be useful to distribute Curator artifacts
as
>>>>>>part of
>>>>>> >> ZooKeeper releases as the preferred/recommended client side
>>>>>>library.
>>>>>> >> Further, at graduation a determination can be made as to
whether
>>>>>>Curator
>>>>>> >> should become a Top Level Project or be merged into ZooKeeper
>>>>>>itself.
>>>>>> >> >
>>>>>> >> > = Known Risks =
>>>>>> >> >
>>>>>> >> > == Orphaned Products ==
>>>>>> >> >
>>>>>> >> > Curator is already deployed in production at multiple
companies
>>>>>>and
>>>>>> they
>>>>>> >> are actively participating in creating new features. Curator
is
>>>>>>getting
>>>>>> >> traction with developers and thus the risks of it being
orphaned
>>>>>>are
>>>>>> >> minimal.
>>>>>> >> >
>>>>>> >> > == Inexperience with Open Source ==
>>>>>> >> >
>>>>>> >> > All code developed for Curator has been open sourced
by Netflix
>>>>>>under
>>>>>> >> Apache 2.0 license.  All committers to Curator are intimately
>>>>>>familiar
>>>>>> with
>>>>>> >> the Apache model for open-source development and are experienced
>>>>>>with
>>>>>> >> working with new contributors.
>>>>>> >> >
>>>>>> >> > == Homogeneous Developers ==
>>>>>> >> >
>>>>>> >> > The initial committers are from a single organization.
However,
>>>>>>we
>>>>>> >> expect that once approved for incubation, the project will
>>>>>>attract new
>>>>>> >> contributors from diverse organizations and will thus grow
>>>>>>organically.
>>>>>> The
>>>>>> >> submission of patches from developers from several different
>>>>>> organizations
>>>>>> >> is a strong indication that Curator will be widely adopted.
>>>>>> >> >
>>>>>> >> > == Reliance on Salaried Developers ==
>>>>>> >> >
>>>>>> >> > It is expected that Curator will be developed on salaried
and
>>>>>> volunteer
>>>>>> >> time, although all of the initial developers will work on
it
>>>>>>mainly on
>>>>>> >> salaried time.
>>>>>> >> >
>>>>>> >> > == Relationships with Other Apache Products ==
>>>>>> >> >
>>>>>> >> > Curator depends upon other Apache Projects: Apache
ZooKeeper,
>>>>>>Apache
>>>>>> >> Log4J, and multiple Apache Commons components. Its build
depends
>>>>>>upon
>>>>>> >> Apache Maven. Notably, there is interest from other Apache
>>>>>>Projects
>>>>>> such as
>>>>>> >> HBase in adopting Curator as the client library for ZooKeeper.
>>>>>>Apache
>>>>>> James
>>>>>> >> Mailbox has already incorporated Curator.
>>>>>> >> >
>>>>>> >> > == An Excessive Fascination with the Apache Brand ==
>>>>>> >> >
>>>>>> >> > We would like Curator to become an Apache project to
further
>>>>>>foster a
>>>>>> >> healthy community of contributors and consumers around the
>>>>>>project.
>>>>>>  Since
>>>>>> >> Curator directly interacts with Apache ZooKeeper and solves
an
>>>>>>important
>>>>>> >> problem of many ZooKeeper users, residing in the Apache
Software
>>>>>> Foundation
>>>>>> >> will increase interaction with the larger community.
>>>>>> >> >
>>>>>> >> > = Documentation =
>>>>>> >> >
>>>>>> >> >  * Curator wiki at GitHub:
>>>>>>https://github.com/Netflix/curator/wiki
>>>>>> >> >  * Curator issues at GitHub:
>>>>>> https://github.com/Netflix/curator/issues
>>>>>> >> >  * Curator javadoc at GitHub:
>>>>>>http://netflix.github.com/curator/doc/
>>>>>> >> >
>>>>>> >> > = Initial Source =
>>>>>> >> >
>>>>>> >> >  * git://github.com/Netflix/curator.git
>>>>>> >> >
>>>>>> >> > == Source and Intellectual Property Submission Plan
==
>>>>>> >> >
>>>>>> >> >  * The initial source is already licensed under the
Apache
>>>>>>License,
>>>>>> >> Version 2.0.
>>>>>>https://github.com/Netflix/curator/blob/master/LICENSE.txt
>>>>>> >> >
>>>>>> >> > == External Dependencies ==
>>>>>> >> >
>>>>>> >> > The required external dependencies are all Apache License
or
>>>>>> compatible
>>>>>> >> licenses. Following components with non-Apache licenses
are
>>>>>>enumerated:
>>>>>> >> >
>>>>>> >> >  * org.slf4j: MIT-like License
>>>>>> >> >  * org.mockito: MIT-like License
>>>>>> >> >
>>>>>> >> > == Cryptography ==
>>>>>> >> >
>>>>>> >> > Curator contains no known cryptography.
>>>>>> >> >
>>>>>> >> > = Required Resources =
>>>>>> >> >
>>>>>> >> > == Mailing lists ==
>>>>>> >> >
>>>>>> >> >  * curator-private (with moderated subscriptions)
>>>>>> >> >  * curator-dev
>>>>>> >> >  * curator-commits
>>>>>> >> >  * curator-user
>>>>>> >> >
>>>>>> >> > == Github Repositories ==
>>>>>> >> >
>>>>>> >> > http://github.com/apache/curator
>>>>>>git://git.apache.org/curator.git
>>>>>> >> >
>>>>>> >> > == Issue Tracking ==
>>>>>> >> >
>>>>>> >> > JIRA Curator (CURATOR)
>>>>>> >> >
>>>>>> >> > == Other Resources ==
>>>>>> >> >
>>>>>> >> > The existing code already has unit and integration
tests so we
>>>>>>would
>>>>>> >> like a Jenkins instance to run them whenever a new patch
is
>>>>>>submitted.
>>>>>> This
>>>>>> >> can be added after project creation.
>>>>>> >> >
>>>>>> >> > = Initial Committers =
>>>>>> >> >
>>>>>> >> >  * Jordan Zimmerman (jzimmerman at netflix dot com)
>>>>>> >> >  * Jay Zarfoss (jzarfoss at netflix dot com)
>>>>>> >> >
>>>>>> >> > = Affiliations =
>>>>>> >> >
>>>>>> >> >  * Jordan Zimmerman, Netflix
>>>>>> >> >  * Jay Zarfoss, Netflix
>>>>>> >> >
>>>>>> >> > = Sponsors =
>>>>>> >> >
>>>>>> >> > == Champion ==
>>>>>> >> >
>>>>>> >> >  * Patrick Hunt
>>>>>> >> >
>>>>>> >> > == Nominated Mentors ==
>>>>>> >> >
>>>>>> >> >  * Patrick Hunt
>>>>>> >> >  * Enis Söztutar
>>>>>> >> >  * Mahadev Konar
>>>>>> >> >
>>>>>> >> > == Sponsoring Entity ==
>>>>>> >> >
>>>>>> >> >  * Apache Incubator PMC
>>>>>> >>
>>>>>> >>
>>>>>>---------------------------------------------------------------------
>>>>>> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>> >> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>> >>
>>>>>> >>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>For additional commands, e-mail: general-help@incubator.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message