incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: [PROPOSAL] Blur for the Apache Incubator
Date Wed, 18 Jul 2012 18:09:00 GMT
Is there anymore feedback before I call the vote?

Thanks!

Aaron

On Sat, Jul 14, 2012 at 11:48 AM, Dave Fisher <dave2wave@comcast.net> wrote:
>
> On Jul 13, 2012, at 2:23 PM, Aaron McCurry wrote:
>
>> Hello!
>>
>> I would like to propose Blur to be an Apache Incubator project.  Blur
>> is a distributed search platform built for low latency searches over
>> large amounts of data.  Blur is scalable and fault tolerant through
>> the use of Hadoop and ZooKeeper.  Thrift is used as the RPC library
>> and the underlying search implementation uses Lucene and the Lucene
>> query syntax.
>>
>> The proposal can be found here:
>> http://wiki.apache.org/incubator/BlurProposal
>>
>> I have included the contexts of the proposal below.
>
> Very cool!
>
> Regards,
> Dave
>
>
>>
>> Thanks!
>> Aaron
>>
>> = Blur Proposal =
>>
>> == Abstract ==
>> Blur is a search platform capable of searching massive amounts of data
>> in a cloud computing environment. Blur leverages several existing
>> Apache projects, including Apache Lucene, Apache Hadoop, Apache
>> !ZooKeeper and Apache Thrift.  Both bulk and near real time (NRT)
>> updates are possible with Blur.  Bulk updates are accomplished using
>> Hadoop Map/Reduce and NRT are performed through direct Thrift calls.
>>
>> == Proposal ==
>> Blur is an open source search platform capable of querying massive
>> amounts of data at incredible speeds. Rather than using the flat,
>> document-like data model used by most search solutions, Blur allows
>> you to build rich data models and search them in a semi-relational
>> manner similar to joins while querying a relational database. Using
>> Blur, you can get precise search results against terabytes of data at
>> Google-like speeds.  Blur leverages multiple open source projects
>> including Hadoop, Lucene, Thrift and !ZooKeeper to create an
>> environment where structured data can be transformed into an index
>> that runs on a Hadoop cluster.  Blur uses the power of Map/Reduce for
>> bulk indexing into Blur.  Server failures are handled automatically by
>> using !ZooKeeper for cluster state and HDFS for index storage.
>>
>> == Background ==
>> Blur was created by Aaron !McCurry in 2010. Blur was developed to
>> solve the challenges in dealing with searching huge quantities of data
>> that the traditional RDBMS solutions could not cope with while still
>> providing JOIN-like capabilities to query the data.  Several other
>> open source projects have implemented aspects of this design including
>> elasticsearch, Katta and Apache Solr.
>>
>> == Rationale ==
>> There is a need for a distributed search capability within the Hadoop
>> ecosystem. Currently, there are no other search solutions that
>> natively leverage HDFS and the failover features of Hadoop in the same
>> manner as the Blur project. The communities we expect to be most
>> interested in such a project are government, health care, and other
>> industries where scalability is a concern. We have made much progress
>> in developing this project over the past 2 years and believe both the
>> project and the interested communities would benefit from this work
>> being openly available and having open development.  In future
>> versions of Blur the API will more closely follow the API’s provided
>> in Lucene so that systems that already use Lucene can more easily
>> scale with Blur. Blur can be viewed as a query execution engine that
>> Lucene based solutions can utilize when scale becomes an issue.
>>
>> == Initial Goals ==
>> The initial goals of the project are:
>> * To migrate the Blur codebase, issue tracking and wiki from
>> github.com and integrate the project with the ASF infrastructure.
>> * Add new committers to the project and grow the community in "The Apache Way".
>>
>> == Current Status ==
>>
>> === Meritocracy ===
>> Blur was initially developed by Aaron !McCurry in June 2010.  Since
>> then Blur has continued to evolve with the support of a small
>> development team at Near Infinity.  As a part of the Apache Software
>> Foundation, the Apache Blur team intends to strongly encourage the
>> community to help with and contribute to the project.  Apache Blur
>> will actively seek potential committers and help them become familiar
>> with the codebase.
>>
>> === Community ===
>> A small community has developed around Blur and several project teams
>> are currently using Blur for their big data search capability. The
>> source code is currently available on GitHub and there is a dedicated
>> website (blur.io) that provides an overview of the project. Blur has
>> been shared with several members of the Apache community and has been
>> presented at the Bay Area HUG (see
>> http://www.meetup.com/hadoop/events/20109471/).
>>
>> === Core Developers ===
>> The current developers are employed by Near Infinity Corporation, but
>> we anticipate interest developing among other companies.
>>
>> === Alignment ===
>> Blur is built on top of a number of Apache projects; Hadoop, Lucene,
>> !ZooKeeper, and Thrift. It builds with Maven.  During the course of
>> Blur development, a couple of patches have been committed back to the
>> Lucene project, including LUCENE-2205 and LUCENE-2215.  Due to the
>> strong relationship with the before mentioned Apache projects, the
>> incubator is a good match for Blur.
>>
>> == Known Risks ==
>>
>> === Orphaned Products ===
>> There is only a small risk of being orphaned. The customers that
>> currently use Blur are committed to improving the codebase of the
>> project due to its fulfilling needs not addressed by any other
>> software. In addition, one customer is providing financial support to
>> further develop Blur given its importance on mission-critical
>> projects.
>>
>> === Inexperience with Open Source ===
>> The codebase has been treated internally as an open source project
>> since its beginning, and Near Infinity has extensive experience
>> developing and releasing open source projects
>> (http://www.nearinfinity.com/products/open_source). We do not
>> anticipate difficulty in operating under the Apache Way.
>>
>> === Homogeneous Developers ===
>> Current developers are all employed by Near Infinity but we are
>> actively seeking contributors from different companies and would
>> welcome their participation.
>>
>> === Reliance on Salaried Developers ===
>> Blur was originally created by Aaron !McCurry as a personal project
>> and he remains the primary contributor.  Currently, Aaron’s employer
>> (Near Infinity) fully supports his continued participation with paid,
>> dedicated time to work on Blur. All other current developers are paid
>> by Near Infinity to work on Blur as well.
>>
>> === Relationships with Other Apache Products ===
>> Blur dependencies:
>>
>> * Apache Hadoop
>> * Apache Lucene
>> * Apache !ZooKeeper
>> * Apache Thrift
>> * Apache log4j
>>
>> === Apache Brand ===
>> Our interest in releasing this code as an Apache project is due to its
>> strong relationship with other Apache projects, i.e. Blur has
>> dependencies on Hadoop, Lucene, !ZooKeeper, and Thrift and its
>> uniqueness within the Hadoop ecosystem.
>>
>> == Documentation ==
>> Current documentation can be found at http://blur.io and
>> https://github.com/nearinfinity/blur.
>>
>> == Initial Source ==
>> Blur has been in development since summer 2010. The core codebase
>> consists of about ~29,000 (~10,000 if the generated RPC code is not
>> included) lines of code mainly Java.
>>
>> == Source and Intellectual Property Submission Plan ==
>> Blur core code, examples, documentation, and training materials will
>> be submitted by Near Infinity Corporation.
>>
>> == External Dependencies ==
>> * concurrentlinkedhashmap - Apache 2.0 License -
>> http://code.google.com/p/concurrentlinkedhashmap/
>>
>> == Cryptography ==
>> none
>>
>> == Required Resources ==
>> * Mailing Lists
>>   * blur-private
>>   * blur-dev
>>   * blur-commits
>>   * blur-user
>> * Subversion Directory
>>   * https://git-wip-us.apache.org/repos/asf/blur.git
>> * Issue Tracking
>>   * JIRA
>> * Continuous Integration
>>   * Jenkins
>> * Web
>>   * http://incubator.apache.org/blur/wiki at http://wiki.apache.org
>> or http://cwiki.apache.org
>>
>> == Initial Committers ==
>> * Aaron !McCurry (aaron.mccurry at nearinfinity dot com)
>> * Scott Leberknight (scott.leberknight at nearinfinity dot com)
>> * Ryan Gimmy (ryan.gimmy at nearinfinity dot com)
>> * Tim Williams (twilliams at apache dot org)
>> * Patrick Hunt (phunt at apache dot org)
>> * Doug Cutting (cutting at apache dot org)
>>
>> == Affiliations ==
>> * Aaron !McCurry, Near Infinity
>> * Scott Leberknight, Near Infinity
>> * Ryan Gimmy, Near Infinity
>> * Patrick Hunt, Cloudera
>> * Doug Cutting, Cloudera
>>
>> == Sponsors ==
>> * Champion: Patrick Hunt
>>
>> == Nominated Mentors ==
>> * Tim Williams  (twilliams at apache dot org)
>> * Doug Cutting (cutting at apache dot org)
>> * Patrick Hunt (phunt at apache dot org)
>>
>> == Sponsoring Entity ==
>> * Apache Incubator
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org


Mime
View raw message