incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Saputra <henry.sapu...@gmail.com>
Subject [PROPOSAL] Apache Spark for the Incubator
Date Fri, 31 May 2013 20:03:36 GMT
I believe it is more of a framework but you can take a look at Shark which
using Spark to do data warehousing that support hive query (
http://shark.cs.berkeley.edu)

- Henry

On Friday, May 31, 2013, Chen, Pei wrote:

> +1 (non-binding)
> This seems like a really interesting project.
> Q- Is Spark just a framework/API or does it also have some tools
> implemented for data analytics?
> --Pei
>
> > -----Original Message-----
> > From: Mattmann, Chris A (398J) [mailto:chris.a.mattmann@jpl.nasa.gov]
> > Sent: Friday, May 31, 2013 2:04 PM
> > To: general@incubator.apache.org
> > Subject: [PROPOSAL] Apache Spark for the Incubator
> >
> > Hi Folks,
> >
> > I'm pleased to bring you a proposal to the Apache Incubator for the
> Apache
> > Spark project: https://wiki.apache.org/incubator/SparkProposal
> >
> > The work originates from the Berkeley AMPLab and through a number of
> > industry participants, and other institutions. Spark is a framework for
> large-
> > scale data analysis on clusters, with a particular focus on low latency
> > operations.
> > The
> > source code is written in Scala, and provides a number of APIs and
> bindings in
> > various programming languages.
> >
> > The proposal text is copied to the bottom of this email. I'm going to
> leave this
> > thread open for the next week for discussion. Once it's died down, I'll
> call an
> > official VOTE.
> >
> > Suresh, Ross G. -- heads up -- this project may be of interest to you
> both and
> > would welcome you guys as additional mentors. We currently have 3
> > mentors committed to the project, but would love to have more. People
> > interested in contributing should declare their interest here on the
> > general@incubator thread and those potential contributors will be
> discussed
> > by the incoming Spark community.
> >
> > Questions -- let's hear em'! :)
> >
> > Cheers,
> > Chris
> > ("Champion", incoming Apache Spark)
> >
> > === Abstract ===
> > Spark is an open source system for large-scale data analysis on clusters.
> >
> > === Proposal ===
> > Spark is an open source system for fast and flexible large-scale data
> analysis.
> > Spark provides a general purpose runtime that supports low-latency
> > execution in several forms. These include interactive exploration of very
> > large datasets, near real-time stream processing, and ad-hoc SQL
> analytics
> > (through higher layer extensions). Spark interfaces with HDFS, HBase,
> > Cassandra and several other storage storage layers, and exposes APIs in
> > Scala, Java and Python.
> > Background
> > Spark started as U.C. Berkeley research project, designed to efficiently
> run
> > machine learning algorithms on large datasets. Over time, it has evolved
> into
> > a general computing engine as outlined above. SparkĀ¹s developer community
> > has also grown to include additional institutions, such as universities,
> > research labs, and corporations. Funding has been provided by various
> > institutions including the U.S. National Science Foundation, DARPA, and a
> > number of industry sponsors. See:
> > https://amplab.cs.berkeley.edu/sponsors/ for full details.
> >
> > === Rationale ===
> > As the number of contributors to Spark has grown, we have sought for a
> > long-term home for the project, and we believe the Apache foundation
> > would be a great fit. Spark is a natural fit for the Apache foundation:
> Spark
> > already interoperates with several existing Apache projects (HDFS, HBase,
> > Hive, Cassandra, Avro and Flume to name a few). The Spark team is
> familiar
> > with the Apache process and and subscribes to the Apache mission - the
> > team includes multiple Apache committers already. Finally, joining Apache
> > will help coordinate the development effort of the growing number of
> > organizations which contribute to Spark.
> >
> > == Initial Goals ==
> > The initial goals will most likely be to move the existing codebase to
> Apache
> > and integrate with the Apache development process. Furthermore, we plan
> > for incremental development, and releases along with the Apache
> > guidelines.
> >
> > === Current Status ===
> > == Meritocracy ==
> > The Spark project already operates on meritocratic principles. Today,
> Spark
> > has several developers and has accepted multiple major patches from
> > outside of U.C. Berkeley. While this process has remained mostly informal
> > (we do not have an official committer list), an implicit organization
> exists in
> > which individuals who contribute major components act as maintainers for
> > those modules. If accepted, the Spark project would include several of
> these
> > participants as committers from the onset. We will work to identify all
> > committers and PPMC members for the project and to operate under the
> > ASF meritocratic principles.
> >
> > === Community ===
> > Acceptance into the Apache foundation would bolster the already strong
> > user and developer community around Spark. That community includes
> > dozens of contributors from several institutions, a meetup group with
> > several hundred members, and an active mailing list composed of hundreds
> > of users.
> > Core Developers
> > The core developers of our project are listed in our contributors and
> initial
> > PPMC below. Though many exist at UC Berkeley, there is a representative
> > cross sampling of other organizations including Quantifind, Microsoft,
> Yahoo!,
> > ClearStory Data, Bizo, Intel, Tagged and Webtrends.
> >
> >
> > === Alignment ===
> > Our proposed ef

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message