Mailing-List: contact general-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: "Mattmann, Chris A (388J)" <chris.a.mattmann@jpl.nasa.gov>
To: "general@lucene.apache.org" <general@lucene.apache.org>
Date: Fri, 2 Jul 2010 10:29:45 -0700
Subject: Re: Draft proposal: Move Lucy to Incubator
Thread-Topic: Draft proposal: Move Lucy to Incubator
Thread-Index: AcsaCjNIF2KQilQ1QQi7zWi1sf/mBQAAfZSM
Message-ID: <C8537019.1774E%Chris.A.Mattmann@jpl.nasa.gov>
In-Reply-To: <20100702171512.GB17879@rectangular.com>
Accept-Language: en-US
Content-Language: en
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C85370191774EChrisAMattmannjplnasagov_"
MIME-Version: 1.0

--_000_C85370191774EChrisAMattmannjplnasagov_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

+1, Marvin. My only comment is that you can hopefully add some additional c=
ommitters when you move into the Incubator to really kick start the project=
.

Cheers,
Chris


On 7/2/10 10:15 AM, "Marvin Humphrey" <marvin@rectangular.com> wrote:


PREFACE
    Lucy is a sub-project which is being spun off from the Lucene TLP but i=
s
    not yet ready for graduation.  We propose to address certain needs of t=
he
    project by assimilating the KinoSearch codebase, and to enter the
    Incubator on a top-level-project track.

ABSTRACT
    Lucy will be a loose port of the Lucene search engine library, written =
in
    C and targeted at dynamic language users.

PROPOSAL
    Lucy has two aims.  First, it will be a high-performance C search engin=
e
    library.  Second, it will maximize its usability and power when accesse=
d
    via dynamic language bindings.  To that end, it will present highly
    idiomatic, carefully tailored APIs for each of its "host" binding
    languages, including support for subclasses written entirely in the "ho=
st"
    language.

BACKGROUND
    Lucy, a "loose C" port of Java Lucene, began as an ambitious, from-scra=
tch
    Lucene sub-project, with David Balmain (author of Ferret, a Ruby/C port=
 of
    Lucene), Doug Cutting, and Marvin Humphrey (founder of KinoSearch, a
    Perl/C port) as committers.  During an initial burst of activity, the
    overall architecture for Lucy was sketched out by Dave and Marvin.
    Unfortunately, Dave became unavailable soon after, and without a workin=
g
    codebase to release or any users, it proved difficult to replace him.
    Still, Marvin carried on their work throughout a period of seemingly lo=
w
    activity.

    In the last year, that work has come to fruition: major technical
    milestones have been achieved and Lucy's underpinnings have been
    completed.  Additionally, other developers from the KinoSearch communit=
y
    have taken an interest in Lucy and have begun to ramp up their
    contributions.  The next steps for Lucy were articulated by the Lucene =
PMC
    in a recent review: make releases, acquire users, grow community.

    To implement the Lucene PMC's recommendations and get to a release as
    quickly as possible, the Lucy community proposes to assimilate the
    KinoSearch codebase, which has been retrofitted to use Lucy's core.  Lu=
cy
    still lacks a number of important indexing and search classes; we wish =
to
    flesh these out via IP clearance work rather than software development.

    Since the Lucene PMC will not be responsible for Lucy much longer, it i=
s
    more appropriate for the software grant to take place within the contex=
t
    of the Incubator than the Lucene TLP.  As none of our current members h=
ave
    Apache PMC experience, we also seek to take advantage of the Incubator
    environment to prepare ourselves for responsible self-governance.

RATIONALE
    There is great hunger for a search engine library in the mode of Lucene
    which is accessible from various dynamic languages, and for one accessi=
ble
    from pure C.  Individuals naturally wish to code in their language of
    choice.  Organizations which do not have significant Java expertise may
    not want to support Java strictly for the sake of running a Lucene
    installation.  Developers may want to take advantage of C's
    interoperability and fine-grained control.  Lucy will meet all these
    demands.

    Apache is a natural home for our project given the way it has always
    operated: user-driven innovation, security as a requirement, lively and
    amiable mailing list discussions, strength through diversity, and so on=
.
    We feel comfortable here, and we believe that we will become exemplary
    Apache citizens.

INITIAL GOALS
    * Make a 1.0 stable release as quickly as possible.
    * Concentrate on community expansion immediately thereafter.
    * Expose a public C API.

CURRENT STATUS
  Meritocracy
    Our initial committer list includes two individuals (Peter Karman and
    Nathan Kurz) who started off as KinoSearch users, demonstrated merit
    through constructive forum participation, adept negotiation, consensus
    building, and submission of high-quality contributions, and were invite=
d
    to become committers.  Peter now rolls most releases.

    We look forward to continuing to operate as a meritocracy under the
    established traditions and rules of the ASF.

  Community
    Lucy's most active participants of late have been drawn from the
    KinoSearch and Lucene communities.  Having been focused on features and
    technical goals for a long time, we are considerably overdue for a stab=
le
    release, and anticipate rapid growth in its wake.

  Core Developers
     * Marvin Humphrey is the project founder of KinoSearch, and co-founded
       the existing Lucy sub-project.  He is presently employed by Eventful=
,
       Inc.
     * Peter Karman has contributed to several open source projects since
       2001, including being a committer at http://swish-e.org/ (a search
       engine), http://code.google.com/p/rose/ (an ORM) and
       http://catalyst.perl.org/ (web framework).  He is employed by Americ=
an
       Public Media.
     * Nathan Kurz has participated in numerous open source projects and ha=
s
       been a KinoSearch committer since 2007.  He is currently Chief Flavo=
r
       Engineer of Scream Sorbet, and writes software in his copious free
       time.

  Alignment
    One Apache value which is particularly cherished by the Lucy community =
is
    codebase transparency.  We have developed institutions which enable us =
to
    measure and maximize usability (see http://wiki.apache.org/lucy/BrainLo=
g),
    and we feel strongly that the bindings for Lucy must present APIs and
    documentation which are idiomatic to the host language culture so that =
end
    users can consume our work as easily as possible.

    The controlled competition of meritocratic community development is als=
o
    very important to us.  There has been substantial cross-pollination of
    ideas between Lucene and Lucy, yielding considerable benefits for both
    projects.  The Lucy developers envision that our host-language
    sub-communities will approach using and extending the library in distin=
ct
    ways; we hope to harness the creative tension between them to drive
    innovation, building productive relationships akin to the one that Luce=
ne
    and Lucy have today.

    A third priority of ours is to be bound by existing Apache institutions=
,
    for the protection of all our stakeholders.

KNOWN RISKS
  Orphaned products
    All initial committers have been associated with the project for severa=
l
    years across multiple jobs.  However, at this time, the project would
    probably not survive the departure of Marvin Humphrey, so there is a ri=
sk
    of being orphaned.  Marvin has no plans to leave, but we have been
    actively working to disperse his knowledge of the code base and
    administrative responsibilities in order to make him dispensable.  Havi=
ng
    staggered badly after Dave Balmain's departure, we are keenly aware of
    this vulnerability and highly motivated to eliminate it.

  Inexperience with Open Source
    The initial committers have all have significant experience with open
    source development, and include one present Apache committer.  We
    recognize that we lack PMC experience and seek to address that deficien=
cy
    by going through the Incubator.  In retrospect, Marvin wishes that Lucy
    had gone through the Incubator during its first inception.

  Homogenous Developers
    Our community is geographically dispersed, with members in San Diego,
    Oakland, and Minneapolis.  We all work for different organizations.

  Reliance on Salaried Developers
    Marvin Humphrey has a great job at Eventful working primarily on this
    project and supporting applications that use it.  Nevertheless, he is
    extremely dedicated to Lucy and is determined to see it through to the
    point where it becomes self-sustaining, regardless of work circumstance=
s.

  Relationships with Other Apache Products
    Lucy's relationship with Lucene of cordial "coopetition" has produced
    benefits for Lucene users in terms of indexing speed, near-real-time
    search support, and more.  We expect this dynamic to continue deliverin=
g
    improvements for all parties involved.

  An Excessive Fascination with the Apache Brand
    Our desire to maintain Lucy's affiliation with Apache has less to do wi=
th
    the brand and more to do with our conviction that developing the projec=
t
    The Apache Way under Apache institutions is in Lucy's best interests.
    However, we have to acknowledge that during its time as a Lucene
    subproject, Lucy has not always fulfilled certain key requirements for =
an
    Apache project.  In particular, it has failed to "release early, releas=
e
    often", and it has made minimal progress in expanding its community.

    We attribute some of our difficulties to the what may have been excess
    ambition in the original Lucy plan, given the scope of the project and =
the
    size of the initial committer list:

        [http://www.apache.org/foundation/how-it-works.html#incubator]

        The basic requirements for incubation are:
          * a working codebase -- over the years and after several
            failures, the foundation came to understand that without an
            initial working codebase, it is generally hard to bootstrap a
            community.

    By rebooting the project with a working codebase, we expect to avoid th=
e
    trap that ensnared Lucy's first incarnation: we will release early,
    release often, accumulate users, nurture contributors, and grow our
    community.

DOCUMENTATION
    * Subversion repository: [http://www.rectangular.com/svn/kinosearch/]
    * Perl API documentation: [http://www.rectangular.com/kinosearch/docs/d=
evel/]
    * Discussion list: [http://www.rectangular.com/mailman/listinfo/kinosea=
rch/]

INITIAL SOURCE
    The initial source will be a snapshot from the KinoSearch subversion
    repository.

SOURCE AND INTELLECTUAL PROPERTY SUBMISSION PLAN
    KinoSearch is currently under a GPL/Artistic license.  There are five
    individuals who have made multiple significant contributions to the
    codebase and whose participation is either essential or would be very
    helpful: Marvin Humphrey, Peter Karman, Nathan Kurz, Chris Nandor, and
    Father Chrysostomos.  All have been contacted and are amenable to
    re-licensing their work and contributing it to Apache.  We will contact=
 as
    many other contributors as possible; if there are any that we cannot
    obtain permission from, we will refactor to expunge their work.

EXTERNAL DEPENDENCIES
    The Perl bindings for KinoSearch currently depend on a few CPAN modules
    which do not have Apache-compatible licenses.  It will be possible to
    eliminate all such dependencies if necessary.

REQUIRED RESOURCES
  Mailing lists
    * lucy-dev
    * lucy-private (with moderated subscriptions)
    * lucy-commits
    * lucy-users

  Subversion Directory
    [http://svn.apache.org/repos/asf/incubator/lucy]

  Issue Tracking
    Lucy already has a JIRA tracker: Lucy (LUCY)

  Other Resources
    Lucy already has a MoinMoin wiki at wiki.apache.org/lucy.  It would be
    convenient to keep it, especially since its current location is also wh=
ere
    it would end up upon TLP graduation, but we will defer to the wishes of
    the Incubator PMC if standard Incubator wiki placement is recommended.

INITIAL COMMITTERS
    1. Marvin Humphrey (marvin at rectangular dot com)
    2. Peter Karman (peter at peknet dot com)
    3. Nathan Kurz ( nate@verse.com )

SPONSORS
  Champion
    TBD

  Nominated Mentors
    TBD

  Sponsoring Entity
    Lucy is currently sponsored by Lucene as a sub-project. This proposal
    advocates changing Lucy's relationship with Apache from developing all =
new
    code as a Lucene sub-project, to instead assimilating existing code
    (KinoSearch) under the sponsorship of the Incubator.


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: Chris.Mattmann@jpl.nasa.gov
WWW:   http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


--_000_C85370191774EChrisAMattmannjplnasagov_--