incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luke Han <luke...@gmail.com>
Subject Re: [VOTE] Accept CarbonData into the Apache Incubator
Date Thu, 26 May 2016 05:43:43 GMT
+1 (binding)


Best Regards!
---------------------

Luke Han

On Wed, May 25, 2016 at 9:44 PM, Wang, Gang1 <gang1.wang@intel.com> wrote:

> +1 (no-binding)
>
> Best Regards
> +Gary.
>
> -----Original Message-----
> From: Cheng, Hao [mailto:hao.cheng@intel.com]
> Sent: Wednesday, May 25, 2016 7:09 PM
> To: general@incubator.apache.org
> Subject: RE: [VOTE] Accept CarbonData into the Apache Incubator
>
> +1
>
> -----Original Message-----
> From: Jacques Nadeau [mailto:jacques@apache.org]
> Sent: Thursday, May 26, 2016 8:26 AM
> To: general@incubator.apache.org
> Subject: Re: [VOTE] Accept CarbonData into the Apache Incubator
>
> +1 (binding)
>
> On Wed, May 25, 2016 at 4:04 PM, John D. Ament <johndament@apache.org>
> wrote:
>
> > +1
> >
> > On Wed, May 25, 2016 at 4:41 PM Jean-Baptiste Onofré <jb@nanthrax.net>
> > wrote:
> >
> > > Hi all,
> > >
> > > following the discussion thread, I'm now calling a vote to accept
> > > CarbonData into the Incubator.
> > >
> > > ​[ ] +1 Accept CarbonData into the Apache Incubator [ ] +0 Abstain [
> > > ] -1 Do not accept CarbonData into the Apache Incubator, because ...
> > >
> > > This vote is open for 72 hours.
> > >
> > > The proposal follows, you can also access the wiki page:
> > > https://wiki.apache.org/incubator/CarbonDataProposal
> > >
> > > Thanks !
> > > Regards
> > > JB
> > >
> > > = Apache CarbonData =
> > >
> > > == Abstract ==
> > >
> > > Apache CarbonData is a new Apache Hadoop native file format for
> > > faster interactive query using advanced columnar storage, index,
> > > compression and encoding techniques to improve computing efficiency,
> > > in turn it will help speedup queries an order of magnitude faster
> > > over PetaBytes of data.
> > >
> > > CarbonData github address:
> > > https://github.com/HuaweiBigData/carbondata
> > >
> > > == Background ==
> > >
> > > Huawei is an ICT solution provider, we are committed to enhancing
> > > customer experiences for telecom carriers, enterprises, and
> > > consumers on big data, In order to satisfy the following customer
> > > requirements, we created a new Hadoop native file format:
> > >
> > >   * Support interactive OLAP-style query over big data in seconds.
> > >   * Support fast query on individual record which require touching
> > > all fields.
> > >   * Fast data loading speed and support incremental load in period
> > > of minutes.
> > >   * Support HDFS so that customer can leverage existing Hadoop cluster.
> > >   * Support time based data retention.
> > >
> > > Based on these requirements, we investigated existing file formats
> > > in the Hadoop eco-system, but we could not find a suitable solution
> > > that satisfying requirements all at the same time, so we start
> > > designing CarbonData.
> > >
> > > == Rationale ==
> > >
> > > CarbonData contains multiple modules, which are classified into two
> > > categories:
> > >
> > >   1. CarbonData File Format: which contains core implementation for
> > > file format such as
> > > columnar,index,dictionary,encoding+compression,API for reading/writing
> etc.
> > >   2. CarbonData integration with big data processing framework such
> > > as Apache Spark, Apache Hive etc. Apache Beam is also planned to
> > > abstract the execution runtime.
> > >
> > > === CarbonData File Format ===
> > >
> > > CarbonData file format is a columnar store in HDFS, it has many
> > > features that a modern columnar format has, such as splittable,
> > > compression schema ,complex data type etc. And CarbonData has
> > > following unique
> > > features:
> > >
> > > ==== Indexing ====
> > >
> > > In order to support fast interactive query, CarbonData leverage
> > > indexing technology to reduce I/O scans. CarbonData files stores
> > > data along with index, the index is not stored separately but the
> > > CarbonData file itself contains the index. In current
> > > implementation, CarbonData supports 3 types of indexing:
> > >
> > > 1. Multi-dimensional Key (B+ Tree index)
> > >   The Data block are written in sequence to the disk and within each
> > > data blocks each column block is written in sequence. Finally, the
> > > metadata block for the file is written with information about byte
> > > positions of each block in the file, Min-Max statistics index and
> > > the start and end MDK of each data block. Since, the entire data in
> > > the file is in sorted order, the start and end MDK of each data
> > > block can be used to construct a B+Tree and the file can be
> > > logically  represented as a
> > > B+Tree with the data blocks as leaf nodes (on disk) and the
> > > B+remaining
> > > non-leaf nodes in memory.
> > > 2. Inverted index
> > >   Inverted index is widely used in search engine. By using this
> > > index, it helps processing/query engine to do filtering inside one
> HDFS block.
> > > Furthermore, query acceleration for count distinct like operation is
> > > made possible when combining bitmap and inverted index in query time.
> > > 3. MinMax index
> > >   For all columns, minmax index is created so that processing/query
> > > engine can skip scan that is not required.
> > >
> > > ==== Global Dictionary ====
> > >
> > > Besides I/O reduction, CarbonData accelerates computation by using
> > > global dictionary, which enables processing/query engines to perform
> > > all processing on encoded data without having to convert the data
> > > (Late Materialization). We have observed dramatic performance
> > > improvement for OLAP analytic scenario where table contains many
> > > columns in string data type. The data is converted back to the user
> > > readable form just before processing/query engine returning results to
> user.
> > >
> > > ==== Column Group ====
> > >
> > > Sometimes users want to perform processing/query on multi-columns in
> > > one table, for example, performing scan for individual record in
> > > troubleshooting scenario. In this case, row format is more efficient
> > > than columnar format since all columns will be touched by the workload.
> > > To accelerate this, CarbonData supports storing a group of column in
> > > row format, so data in column group is stored together and enable
> > > fast retrieval.
> > >
> > > ==== Optimized for multiple use cases ====
> > >
> > > CarbonData indices and dictionary is highly configurable. To make
> > > storage optimized for different use cases, user can configure what
> > > to index, so user can decide and tune the format before loading data
> > > into CarbonData.
> > >
> > > For example
> > >
> > > || Use Case || Supporting Features || Interactive OLAP query ||
> > > || Columnar format, Multi-dimensional Key (B+
> > > Tree index), Minmax index, Inverted index ||
> > > || High throughput scan || Global dictionary, Minmax index || Low
> > > || latency point query || Multi-dimensional Key (B+ Tree index),
> > > Partitioning ||
> > > || Individual record query || Column group, Global dictionary ||
> > >
> > > === BigData Processing Framework Integration ===
> > >
> > >   * CarbonData provides InputFormat/OutputFormat interfaces for
> > > Reading/Writing data from the CarbonData files and at the same time
> > > provides abstract API for processing data stored as Carbondata
> > > format with data processing framework.
> > >   * CarbonData provides deep integration with Apache Spark including
> > > predicate push down, column pruning, aggregation push down etc. So
> > > users can use Spark SQL to connect and query from CarbonData.
> > >   * CarbonData can integrate with various big data Query/Processing
> > > framework on Hadoop eco-system such as Apache Spark,Apache Hive etc.
> > >
> > > Example:
> > >
> > >
> > https://github.com/HuaweiBigData/carbondata/blob/master/examples/src/m
> > ain/scala/org/carbondata/examples/CarbonExample.scala
> > >
> > > == Initial Goals ==
> > >
> > > Our initial goals are to bring CarbonData into the ASF, transition
> > > internal engineering processes into the open, and foster a
> > > collaborative development model according to the "Apache Way".
> > >
> > > == Current Status ==
> > >
> > > CarbonData is production ready and already provide a large set of
> > features.
> > > The current license is already Apache 2.0.
> > >
> > > == Meritocracy ==
> > >
> > > We intend to radically expand the initial developer and user
> > > community by running the project in accordance with the "Apache
> > > Way". Users and new contributors will be treated with respect and
> > > welcomed. By participating in the community and providing quality
> > > patches/support that move the project forward, they will earn merit.
> > > They also will be encouraged to provide non-code contributions
> > > (documentation, events, community management, etc.) and will gain
> > > merit for doing so. Those with a proven support and quality track
> > > record will be encouraged to become committers.
> > >
> > > == Community ==
> > >
> > > If CarbonData is accepted for incubation, the primary initial goal
> > > is to build a large community. We really trust that CarbonData will
> > > become a key project for big data column-like platforms, and so, we
> > > bet on a large community of users and developers.
> > >
> > > == Known Risks ==
> > >
> > > Development has been sponsored mostly by a one company.For the
> > > project to fully transition to the Apache Way governance model,
> > > development must shift towards the meritocracy-centric model of
> > > growing a community of contributors balanced with the needs for
> > > extreme stability and core implementation coherency.
> > >
> > > == Orphaned products ==
> > >
> > > Huawei is fully committed CarbonData. Moreover, Huawei has a vested
> > > interest in making CarbonData succeed by driving its close
> > > integration with sister ASF projects. We expect this to further
> > > reduces the risk of orphaning the product.
> > >
> > > == Inexperience with Open Source ==
> > >
> > > Huawei has been developing and using open source software since a
> > > long time. Additionally, several ASF veterans agreed to mentor the
> > > project and are listed in this proposal. The project will rely on
> > > their guidance and collective wisdom to quickly transition the
> > > entire team of initial committers towards practicing the Apache Way.
> > >
> > > == Reliance on Salaried Developers ==
> > >
> > > Most of the contributors are paid to work in big data space. While
> > > they might wander from their current employers, they are unlikely to
> > > venture far from their core expertises and thus will continue to be
> > > engaged with the project regardless of their current employers.
> > >
> > > == An Excessive Fascination with the Apache Brand ==
> > >
> > > While we intend to leverage the Apache ‘branding’ when talking to
> > > other projects as testament of our project’s ‘neutrality’, we have
> > > no plans for making use of Apache brand in press releases nor
> > > posting billboards advertising acceptance of CarbonData into Apache
> Incubator.
> > >
> > > == Initial Source ==
> > >
> > > https://github.com/HuaweiBigData/carbondata.git
> > >
> > > == External Dependencies ==
> > >
> > > All external dependencies are licensed under an Apache 2.0 license
> > > or Apache-compatible license. As we grow the Carbondata community we
> > > will configure our build process to require and validate all
> > > contributions and dependencies are licensed under the Apache 2.0
> > > license or are under an Apache-compatible license.
> > >
> > >   * Apache Spark
> > >   * Apache Hadoop
> > >   * Apache Maven
> > >   * Apache Commons
> > >   * Apache Log4j
> > >   * Apache Thrift
> > >   * Apache Zookeeper
> > >   * Scala
> > >   * Snappy
> > >   * Kettle (Pentaho)
> > >   * Eigenbase
> > >   * Fastutil
> > >   * GSON
> > >   * Jmockit
> > >   * Junit
> > >
> > > == Required Resources ==
> > >
> > > === Mailing lists ===
> > >
> > >   * private@carbondata.incubator.apache.org (moderated subscriptions)
> > >   * commits@carbondata.incubator.apache.org
> > >   * dev@carbondata.incubator.apache.org
> > >   * issues@carbondata.incubator.apache.org
> > >
> > > === Git Repository ===
> > >
> > >   * https://git-wip-us.apache.org/repos/asf/incubator-carbondata.git
> > >
> > > === Issue Tracking ===
> > >
> > >   * JIRA Project CarbonData (CarbonData)
> > >
> > > === Initial Committers ===
> > >
> > >   * Liang Chenliang
> > >   * Jean-Baptiste Onofré
> > >   * Henry Saputra
> > >   * Uma Maheswara Rao G
> > >   * Jenny MA
> > >   * Jacky Likun
> > >   * Vimal Das Kammath
> > >   * Jarray Qiuheng
> > >
> > > === Affiliations ===
> > >
> > >   * Huawei: Liang Chenliang
> > >   * Talend: Jean-Baptiste Onofré
> > >   * Ebay: Henry Saputra
> > >   * Intel: Uma Maheswara Rao G
> > >
> > > === Sponsors ===
> > >
> > > === Champion ===
> > >
> > >   * Jean-Baptiste Onofré - Apache Member
> > >
> > > === Mentors ===
> > >
> > >   * Henry Saputra (eBay)
> > >   * Jean-Baptiste Onofré (Talend)
> > >   * Uma Maheswara Rao G (Intel)
> > >
> > > === Sponsoring Entity ===
> > >
> > > The Apache Incubator
> > >
> > > --------------------------------------------------------------------
> > > - To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > > For additional commands, e-mail: general-help@incubator.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message