incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jake Farrell <jfarr...@apache.org>
Subject Re: [PROPOSAL] Parquet
Date Sat, 17 May 2014 02:40:40 GMT
Thanks Dmitriy
Thats what I thought we would do but just wanted to make sure, proposal
updated to reflect the main imports that we will need

-Jake


On Fri, May 16, 2014 at 6:55 PM, Dmitriy Ryaboy <dvryaboy@gmail.com> wrote:

> Roman -- you've been added, thanks for volunteering :)
>
> One Jake's repo question, I think it would be somewhat problematic, and
> we'd prefer multiple git repos. Format and the java code are currently
> versioned separately and have different builds (format is much lighter).
> Plus, if we add, for example, a general-use C library for reading Parquet
> files, we wouldn't want to conflate its build with the mvn-based builds for
> parquet-mr.
>
>
>
>
> On Fri, May 16, 2014 at 11:56 AM, Roman Shaposhnik <rvs@apache.org> wrote:
>
> > Hi!
> >
> > proposal looks good to me and I am very much looking
> > for a voting thread.
> >
> > One small request, since I plan to spend a fair amount
> > of time on Parquet anyway, would you guys be ok
> > with adding me as an extra mentor so I can help
> > with that aspect of the project as well?
> >
> > Thanks,
> > Roman.
> >
> > P.S. Plus it has an added benefit of increasing diversity
> > of affiliations from the get go.
> >
> > On Mon, May 12, 2014 at 10:02 AM, Chris Aniszczyk <caniszczyk@gmail.com>
> > wrote:
> > > We would like to propose Parquet as an Apache Incubator project.
> > > https://wiki.apache.org/incubator/ParquetProposal
> > >
> > > Feel free to comment, we'll go for a vote in a week or two or whenever
> > > consensus has been reached on the proposal.
> > >
> > > I've posted posted the text of the proposal below:
> > >
> > > == Abstract ==
> > > Parquet is a columnar storage format for Hadoop.
> > >
> > > == Proposal ==
> > >
> > > We created Parquet to make the advantages of compressed, efficient
> > columnar
> > > data representation available to any project in the Hadoop ecosystem,
> > > regardless of the choice of data processing framework, data model, or
> > > programming language.
> > >
> > > == Background ==
> > >
> > > Parquet is built from the ground up with complex nested data structures
> > in
> > > mind, and uses the repetition/definition level approach to encoding
> such
> > > data structures, as popularized by Google Dremel (
> > > https://blog.twitter.com/2013/dremel-made-simple-with-parquet). We
> > believe
> > > this approach is superior to simple flattening of nested name spaces.
> > >
> > > Parquet is built to support very efficient compression and encoding
> > > schemes. Parquet allows compression schemes to be specified on a
> > per-column
> > > level, and is future-proofed to allow adding more encodings as they are
> > > invented and implemented. We separate the concepts of encoding and
> > > compression, allowing parquet consumers to implement operators that
> work
> > > directly on encoded data without paying decompression and decoding
> > penalty
> > > when possible.
> > >
> > > == Rationale ==
> > >
> > > Parquet is built to be used by anyone. We believe that an efficient,
> > > well-implemented columnar storage substrate should be useful to all
> > > frameworks without the cost of extensive and difficult to set up
> > > dependencies.
> > >
> > > Furthermore, the rapid growth of Parquet community is empowered by open
> > > source. We believe the Apache foundation is a great fit as the
> long-term
> > > home for Parquet, as it provides an established process for
> > > community-driven development and decision making by consensus. This is
> > > exactly the model we want for future Parquet development.
> > >
> > > == Initial Goals ==
> > >
> > > * Move the existing codebase to Apache
> > > * Integrate with the Apache development process
> > > * Ensure all dependencies are compliant with Apache License version 2.0
> > > * Incremental development and releases per Apache guidelines
> > >
> > > == Current Status ==
> > >
> > > Parquet has undergone 2 major releases:
> > > https://github.com/Parquet/parquet-format/releases of the core format
> > and
> > > 22 releases: https://github.com/Parquet/parquet-mr/releases of the
> > > supporting set of Java libraries.
> > >
> > > The Parquet source is currently hosted at GitHub, which will seed the
> > > Apache git repository.
> > >
> > > === Meritocracy ===
> > >
> > > We plan to invest in supporting a meritocracy. We will discuss the
> > > requirements in an open forum. Several companies have already expressed
> > > interest in this project, and we intend to invite additional developers
> > to
> > > participate. We will encourage and monitor community participation so
> > that
> > > privileges can be extended to those that contribute.
> > >
> > > === Community ===
> > >
> > > There is a large need for an advanced columnar storage format for
> Hadoop.
> > > Parquet is being used in production by many organizations (see
> > > https://github.com/Parquet/parquet-mr/blob/master/PoweredBy.md)
> > >
> > >  * Cloudera: https://twitter.com/HenryR/statuses/324222874011451392
> > >  * Criteo: https://twitter.com/julsimon/statuses/312114074911666177
> > >  * Salesforce:
> > https://twitter.com/TwitterOSS/statuses/392734610116726784
> > >  * Stripe: https://twitter.com/avibryant/statuses/391339949250715648
> > >  * Twitter: https://twitter.com/J_/statuses/315844725611581441
> > >
> > > By bringing Parquet into Apache, we believe that the community will
> grow
> > > even bigger.
> > >
> > > === Core Developers ===
> > >
> > > Parquet was initially developed as a collaboration between Twitter,
> > > Cloudera and Criteo.
> > >
> > > See
> > >
> >
> https://blog.twitter.com/2013/announcing-parquet-10-columnar-storage-for-hadoop
> > >
> > > === Alignment ===
> > >
> > > We believe that having Parquet at Apache will help further the growth
> of
> > > the big-data community, as it will encourage cooperation within the
> > greater
> > > ecosystem of projects spawned by Apache Hadoop. The alignment is also
> > > beneficial to other Apache communities (such as Hadoop, Hive, Avro).
> > >
> > > == Known Risks ==
> > >
> > > === Orphaned Products ===
> > >
> > > The risk of the Parquet project being abandoned is minimal. There are
> > many
> > > organizations using Parquet in production, including Twitter, Cloudera,
> > > Stripe, and Salesforce (
> > > http://blog.cloudera.com/blog/2013/10/parquet-at-salesforce-com/).
> > >
> > > === Inexperience with Open Source ===
> > >
> > > Parquet has existed as a healthy open source for one year. During that
> > > time, we have curated an open-source community successfully, attracting
> > > over 40 contributors (see
> > > https://github.com/Parquet/parquet-mr/graphs/contributors) from a
> > diverse
> > > group of companies.
> > > Several of the core contributors to the project are deeply familiar
> with
> > > OSS and Apache specifically: Julien Le Dem is the current PMC Chair for
> > > Apache Pig, and Dmitriy Ryaboy, Aniket Mokashi, and Jonathan Coveney
> are
> > > also Apache Pig committers with contributions to several other Apache
> > > projects. Todd Lipcon and Tom White are committers to Apache Hadoop and
> > > multiple other related projects. Brock Noland is a Hive committer.
> > >
> > > === Homogenous Developers ===
> > >
> > > The initial committers come from a number of companies and countries.
> > > Parquet has an active community of developers, and we are committed to
> > > recruiting additional committers based on their contributions to the
> > > project. The java library component alone has contributions from 31
> > > individual github accounts, 14 of which contributed over 1000 lines of
> > code.
> > >
> > > === Reliance on Salaried Developers ===
> > >
> > > It is expected that Parquet development will occur on both salaried
> time
> > > and on volunteer time, after hours. The majority of initial committers
> > are
> > > paid by their employers to contribute to this project. However, they
> are
> > > all passionate about the project, and we are confident that the project
> > > will continue even if no salaried developers contribute to the project.
> > As
> > > evidence of this statement, we present the GitHub punchcard (see
> > > https://github.com/Parquet/parquet-mr/graphs/punch-card) showing that
> a
> > lot
> > > of activity happens on weekends. We are committed to recruiting
> > additional
> > > committers including non-salaried developers.
> > >
> > > === Relationships with Other Apache Products ===
> > >
> > > As mentioned in the Alignment section, Parquet is closely related to
> > > Hadoop, Pig, Avro, Thrift, YARN and Mesos in a numerous ways. We look
> > > forward to collaborating with those communities, as well as other
> Apache
> > > communities (including Apache S4 which focuses on stateful low-latency
> > > processing).
> > >
> > > === An Excessive Fascination with the Apache Brand ===
> > >
> > > Parquet is an already healthy and well known open source project. This
> > > proposal is not for the purpose of generating publicity. Rather, the
> > > primary benefits to joining Apache are those outlined in the Rationale
> > > section.
> > >
> > > == Documentation ==
> > >
> > > Documentation is currently located as README markdown files:
> > >
> > > * https://github.com/Parquet/parquet-format
> > > * https://github.com/Parquet/parquet-mr
> > >
> > > == Source and Intellectual Property Submission Plan ==
> > >
> > > The Parquet codebase is currently hosted on Github:
> > > https://github.com/Parquet.
> > >
> > > This is the exact codebase that we would migrate to the Apache
> > foundation.
> > >
> > > == External Dependencies ==
> > >
> > >  * Junit: EPL
> > >  * Apache Commons: ALv2
> > >  * Apache Thrift: ALv2
> > >  * Apache Maven: ALv2
> > >  * Apache Avro: ALv2
> > >  * Apache Hadoop: ALv2
> > >  * Google Guava: ALv2
> > >
> > > == Cryptography ==
> > >
> > > We do not expect Parquet to be a controlled export item due to the use
> of
> > > encryption.
> > >
> > > == Required Resources ==
> > >
> > > === Mailing lists ===
> > >
> > >  * parquet-dev
> > >  * parquet-user
> > >
> > > == Subversion Directory ==
> > >
> > > Git is the preferred source control system: git://
> git.apache.org/parquet
> > >
> > > == Issue Tracking ==
> > >
> > > JIRA: Parquet (PARQUET)
> > >
> > > == Initial Committers ==
> > >
> > >  * Aniket Mokashi
> > >  * Brock Noland
> > >  * Chris Aniszczyk <zx@twitter.com>
> > >  * Dmitriy Ryaboy <dmitriy@twitter.com>
> > >  * Jake Farrell
> > >  * Julien Le Dem <julien@apache.org>
> > >  * Lukas Nalezenec
> > >  * Marcel Kornacker
> > >  * Mickael Lacour
> > >  * Nong Li
> > >  * Remy Pecqueur
> > >  * Tianshuo Deng
> > >  * Tom White
> > >
> > > == Affiliations ==
> > >
> > >  * Aniket Mokashi - Twitter
> > >  * Brock Noland - Cloudera
> > >  * Chris Aniszczyk - Twitter
> > >  * Dmitriy Ryaboy - Twitter
> > >  * Jake Farrell
> > >  * Julien Le Dem - Twitter
> > >  * Lukas Nalezenec
> > >  * Marcel Kornacker - Cloudera
> > >  * Mickael Lacour - Criteo
> > >  * Nong Li - Cloudera
> > >  * Remy Pecqueur - Criteo
> > >  * Tianshuo Deng - Twitter
> > >  * Tom White - Cloudera
> > >
> > > == Sponsors ==
> > >
> > > === Champion ===
> > >
> > >  * Todd Lipcon
> > >
> > > === Nominated Mentors ===
> > >
> > >  * Tom White
> > >  * Chris Mattmann
> > >  * Jake Farrell
> > >
> > > === Sponsoring Entity ===
> > >
> > > The Apache Incubator
> > >
> > > --
> > > Cheers,
> > >
> > > Chris Aniszczyk
> > > http://aniszczyk.org
> > > +1 512 961 6719
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message