incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naresh Agarwal <naresh.agar...@gmail.com>
Subject Re: [VOTE] Accept Beam into the Apache Incubator
Date Fri, 29 Jan 2016 01:34:42 GMT
+1  (non-binding)

Thanks
Naresh
On 29 Jan 2016 06:18, "Hadrian Zbarcea" <hzbarcea@gmail.com> wrote:

> +1 (binding)
>
> Man, congrats on a job fantastically well done. This is ASF incubator
> participation at its best.
>
> Expectations are high now. I am looking forward to exemplary governance
> and speedy graduation.
>
> Best of luck,
> Hadrian
>
> On 01/28/2016 09:28 AM, Jean-Baptiste Onofré wrote:
>
>> Hi,
>>
>> the Beam proposal (initially Dataflow) was proposed last week.
>>
>> The complete discussion thread is available here:
>>
>>
>> http://mail-archives.apache.org/mod_mbox/incubator-general/201601.mbox/%3CCA%2B%3DKJmvj4wyosNTXVpnsH8PhS7jEyzkZngc682rGgZ3p28L42Q%40mail.gmail.com%3E
>>
>>
>> As reminder the BeamProposal is here:
>>
>> https://wiki.apache.org/incubator/BeamProposal
>>
>> Regarding all the great feedbacks we received on the mailing list, we
>> think it's time to call a vote to accept Beam into the Incubator.
>>
>> Please cast your vote to:
>> [] +1 - accept Apache Beam as a new incubating project
>> []  0 - not sure
>> [] -1 - do not accept the Apache Beam project (because: ...)
>>
>> Thanks,
>> Regards
>> JB
>> ----
>> ## page was renamed from DataflowProposal
>> = Apache Beam =
>>
>> == Abstract ==
>>
>> Apache Beam is an open source, unified model and set of
>> language-specific SDKs for defining and executing data processing
>> workflows, and also data ingestion and integration flows, supporting
>> Enterprise Integration Patterns (EIPs) and Domain Specific Languages
>> (DSLs). Dataflow pipelines simplify the mechanics of large-scale batch
>> and streaming data processing and can run on a number of runtimes like
>> Apache Flink, Apache Spark, and Google Cloud Dataflow (a cloud service).
>> Beam also brings DSL in different languages, allowing users to easily
>> implement their data integration processes.
>>
>> == Proposal ==
>>
>> Beam is a simple, flexible, and powerful system for distributed data
>> processing at any scale. Beam provides a unified programming model, a
>> software development kit to define and construct data processing
>> pipelines, and runners to execute Beam pipelines in several runtime
>> engines, like Apache Spark, Apache Flink, or Google Cloud Dataflow. Beam
>> can be used for a variety of streaming or batch data processing goals
>> including ETL, stream analysis, and aggregate computation. The
>> underlying programming model for Beam provides MapReduce-like
>> parallelism, combined with support for powerful data windowing, and
>> fine-grained correctness control.
>>
>> == Background ==
>>
>> Beam started as a set of Google projects (Google Cloud Dataflow) focused
>> on making data processing easier, faster, and less costly. The Beam
>> model is a successor to MapReduce, FlumeJava, and Millwheel inside
>> Google and is focused on providing a unified solution for batch and
>> stream processing. These projects on which Beam is based have been
>> published in several papers made available to the public:
>>
>>   * MapReduce - http://research.google.com/archive/mapreduce.html
>>   * Dataflow model  - http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
>>   * FlumeJava - http://research.google.com/pubs/pub35650.html
>>   * MillWheel - http://research.google.com/pubs/pub41378.html
>>
>> Beam was designed from the start to provide a portable programming
>> layer. When you define a data processing pipeline with the Beam model,
>> you are creating a job which is capable of being processed by any number
>> of Beam processing engines. Several engines have been developed to run
>> Beam pipelines in other open source runtimes, including a Beam runner
>> for Apache Flink and Apache Spark. There is also a “direct runner”, for
>> execution on the developer machine (mainly for dev/debug purposes).
>> Another runner allows a Beam program to run on a managed service, Google
>> Cloud Dataflow, in Google Cloud Platform. The Dataflow Java SDK is
>> already available on GitHub, and independent from the Google Cloud
>> Dataflow service. Another Python SDK is currently in active development.
>>
>> In this proposal, the Beam SDKs, model, and a set of runners will be
>> submitted as an OSS project under the ASF. The runners which are a part
>> of this proposal include those for Spark (from Cloudera), Flink (from
>> data Artisans), and local development (from Google); the Google Cloud
>> Dataflow service runner is not included in this proposal. Further
>> references to Beam will refer to the Dataflow model, SDKs, and runners
>> which are a part of this proposal (Apache Beam) only. The initial
>> submission will contain the already-released Java SDK; Google intends to
>> submit the Python SDK later in the incubation process. The Google Cloud
>> Dataflow service will continue to be one of many runners for Beam, built
>> on Google Cloud Platform, to run Beam pipelines. Necessarily, Cloud
>> Dataflow will develop against the Apache project additions, updates, and
>> changes. Google Cloud Dataflow will become one user of Apache Beam and
>> will participate in the project openly and publicly.
>>
>> The Beam programming model has been designed with simplicity,
>> scalability, and speed as key tenants. In the Beam model, you only need
>> to think about four top-level concepts when constructing your data
>> processing job:
>>
>>   * Pipelines - The data processing job made of a series of computations
>> including input, processing, and output
>>   * PCollections - Bounded (or unbounded) datasets which represent the
>> input, intermediate and output data in pipelines
>>   * PTransforms - A data processing step in a pipeline in which one or
>> more PCollections are an input and output
>>   * I/O Sources and Sinks - APIs for reading and writing data which are
>> the roots and endpoints of the pipeline
>>
>> == Rationale ==
>>
>> With Google Dataflow, Google intended to develop a framework which
>> allowed developers to be maximally productive in defining the
>> processing, and then be able to execute the program at various levels of
>> latency/cost/completeness without re-architecting or re-writing it. This
>> goal was informed by Google’s past experience  developing several
>> models, frameworks, and tools useful for large-scale and distributed
>> data processing. While Google has previously published papers describing
>> some of its technologies, Google decided to take a different approach
>> with Dataflow. Google open-sourced the SDK and model alongside
>> commercialization of the idea and ahead of publishing papers on the
>> topic. As a result, a number of open source runtimes exist for Dataflow,
>> such as the Apache Flink and Apache Spark runners.
>>
>> We believe that submitting Beam as an Apache project will provide an
>> immediate, worthwhile, and substantial contribution to the open source
>> community. As an incubating project, we believe Dataflow will have a
>> better opportunity to provide a meaningful contribution to OSS and also
>> integrate with other Apache projects.
>>
>> In the long term, we believe Beam can be a powerful abstraction layer
>> for data processing. By providing an abstraction layer for data
>> pipelines and processing, data workflows can be increasingly portable,
>> resilient to breaking changes in tooling, and compatible across many
>> execution engines, runtimes, and open source projects.
>>
>> == Initial Goals ==
>>
>> We are breaking our initial goals into immediate (< 2 months),
>> short-term (2-4 months), and intermediate-term (> 4 months).
>>
>> Our immediate goals include the following:
>>
>>   * Plan for reconciling the Dataflow Java SDK and various runners into
>> one project
>>   * Plan for refactoring the existing Java SDK for better extensibility
>> by SDK and runner writers
>>   * Validating all dependencies are ASL 2.0 or compatible
>>   * Understanding and adapting to the Apache development process
>>
>> Our short-term goals include:
>>
>>   * Moving the newly-merged lists, and build utilities to Apache
>>   * Start refactoring codebase and move code to Apache Git repo
>>   * Continue development of new features, functions, and fixes in the
>> Dataflow Java SDK, and Dataflow runners
>>   * Cleaning up the Dataflow SDK sources and crafting a roadmap and plan
>> for how to include new major ideas, modules, and runtimes
>>   * Establishment of easy and clear build/test framework for Dataflow
>> and associated runtimes; creation of testing, rollback, and validation
>> policy
>>   * Analysis and design for work needed to make Beam a better data
>> processing abstraction layer for multiple open source frameworks and
>> environments
>>
>> Finally, we have a number of intermediate-term goals:
>>
>>   * Roadmapping, planning, and execution of integrations with other OSS
>> and non-OSS projects/products
>>   * Inclusion of additional SDK for Python, which is under active
>> development
>>
>> == Current Status ==
>>
>> === Meritocracy ===
>>
>> Dataflow was initially developed based on ideas from many employees
>> within Google. As an ASL OSS project on GitHub, the Dataflow SDK has
>> received contributions from data Artisans, Cloudera Labs, and other
>> individual developers. As a project under incubation, we are committed
>> to expanding our effort to build an environment which supports a
>> meritocracy. We are focused on engaging the community and other related
>> projects for support and contributions. Moreover, we are committed to
>> ensure contributors and committers to Dataflow come from a broad mix of
>> organizations through a merit-based decision process during incubation.
>> We believe strongly in the Beam model and are committed to growing an
>> inclusive community of Beam contributors.
>>
>> === Community ===
>>
>> The core of the Dataflow Java SDK has been developed by Google for use
>> with Google Cloud Dataflow. Google has active community engagement in
>> the SDK GitHub repository
>> (https://github.com/GoogleCloudPlatform/DataflowJavaSDK), on Stack
>> Overflow
>> (http://stackoverflow.com/questions/tagged/google-cloud-dataflow) and
>> has had contributions from a number of organizations and indivuduals.
>>
>> Everyday, Cloud Dataflow is actively used by a number of organizations
>> and institutions for batch and stream processing of data. We believe
>> acceptance will allow us to consolidate existing Dataflow-related work,
>> grow the Dataflow community, and deepen connections between Dataflow and
>> other open source projects.
>>
>> === Core Developers ===
>>
>> The core developers for Dataflow and the Dataflow runners are:
>>
>>   * Frances Perry
>>   * Tyler Akidau
>>   * Davor Bonaci
>>   * Luke Cwik
>>   * Ben Chambers
>>   * Kenn Knowles
>>   * Dan Halperin
>>   * Daniel Mills
>>   * Mark Shields
>>   * Craig Chambers
>>   * Maximilian Michels
>>   * Tom White
>>   * Josh Wills
>>   * Robert Bradshaw
>>
>> === Alignment ===
>>
>> The Beam SDK can be used to create Beam pipelines which can be executed
>> on Apache Spark or Apache Flink. Beam is also related to other Apache
>> projects, such as Apache Crunch. We plan on expanding functionality for
>> Beam runners, support for additional domain specific languages, and
>> increased portability so Beam is a powerful abstraction layer for data
>> processing.
>>
>> == Known Risks ==
>>
>> === Orphaned Products ===
>>
>> The Dataflow SDK is presently used by several organizations, from small
>> startups to Fortune 100 companies, to construct production pipelines
>> which are executed in Google Cloud Dataflow. Google has a long-term
>> commitment to advance the Dataflow SDK; moreover, Dataflow is seeing
>> increasing interest, development, and adoption from organizations
>> outside of Google.
>>
>> === Inexperience with Open Source ===
>>
>> Google believes strongly in open source and the exchange of information
>> to advance new ideas and work. Examples of this commitment are active
>> OSS projects such as Chromium (https://www.chromium.org) and Kubernetes
>> (http://kubernetes.io/). With Dataflow, we have tried to be increasingly
>> open and forward-looking; we have published a paper in the VLDB
>> conference describing the Dataflow model
>> (http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to
>> release the Dataflow SDK as open source software with the launch of
>> Cloud Dataflow. Our submission to the Apache Software Foundation is a
>> logical extension of our commitment to open source software.
>>
>> === Homogeneous Developers ===
>>
>> The majority of committers in this proposal belong to Google due to the
>> fact that Dataflow has emerged from several internal Google projects.
>> This proposal also includes committers outside of Google who are
>> actively involved with other Apache projects, such as Hadoop, Flink, and
>> Spark.  We expect our entry into incubation will allow us to expand the
>> number of individuals and organizations participating in Dataflow
>> development. Additionally, separation of the Dataflow SDK from Google
>> Cloud Dataflow allows us to focus on the open source SDK and model and
>> do what is best for this project.
>>
>> === Reliance on Salaried Developers ===
>>
>> The Dataflow SDK and Dataflow runners have been developed primarily by
>> salaried developers supporting the Google Cloud Dataflow project. While
>> the Dataflow SDK and Cloud Dataflow have been developed by different
>> teams (and this proposal would reinforce that separation) we expect our
>> initial set of developers will still primarily be salaried. Contribution
>> has not been exclusively from salaried developers, however. For example,
>> the contrib directory of the Dataflow SDK
>> (
>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib
>> )
>> contains items from free-time contributors. Moreover, seperate projects,
>> such as ScalaFlow (https://github.com/darkjh/scalaflow) have been
>> created around the Dataflow model and SDK. We expect our reliance on
>> salaried developers will decrease over time during incubation.
>>
>> === Relationship with other Apache products ===
>>
>> Dataflow directly interoperates with or utilizes several existing Apache
>> projects.
>>
>>   * Build
>>    * Apache Maven
>>   * Data I/O, Libraries
>>    * Apache Avro
>>    * Apache Commons
>>   * Dataflow runners
>>    * Apache Flink
>>    * Apache Spark
>>
>> Beam when used in batch mode shares similarities with Apache Crunch;
>> however, Beam is focused on a model, SDK, and abstraction layer beyond
>> Spark and Hadoop (MapReduce.) One key goal of Beam is to provide an
>> intermediate abstraction layer which can easily be implemented and
>> utilized across several different processing frameworks.
>>
>> === An excessive fascination with the Apache brand ===
>>
>> With this proposal we are not seeking attention or publicity. Rather, we
>> firmly believe in the Beam model, SDK, and the ability to make Beam a
>> powerful yet simple framework for data processing. While the Dataflow
>> SDK and model have been open source, we believe putting code on GitHub
>> can only go so far. We see the Apache community, processes, and mission
>> as critical for ensuring the Beam SDK and model are truly
>> community-driven, positively impactful, and innovative open source
>> software. While Google has taken a number of steps to advance its
>> various open source projects, we believe Beam is a great fit for the
>> Apache Software Foundation due to its focus on data processing and its
>> relationships to existing ASF projects.
>>
>> == Documentation ==
>>
>> The following documentation is relevant to this proposal. Relevant
>> portion of the documentation will be contributed to the Apache Beam
>> project.
>>
>>   * Dataflow website: https://cloud.google.com/dataflow
>>   * Dataflow programming model:
>> https://cloud.google.com/dataflow/model/programming-model
>>   * Codebases
>>    * Dataflow Java SDK:
>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK
>>    * Flink Dataflow runner:
>> https://github.com/dataArtisans/flink-dataflow
>>    * Spark Dataflow runner: https://github.com/cloudera/spark-dataflow
>>   * Dataflow Java SDK issue tracker:
>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues
>>   * google-cloud-dataflow tag on Stack Overflow:
>> http://stackoverflow.com/questions/tagged/google-cloud-dataflow
>>
>> == Initial Source ==
>>
>> The initial source for Beam which we will submit to the Apache
>> Foundation will include several related projects which are currently
>> hosted on the GitHub repositories:
>>
>>   * Dataflow Java SDK
>> (https://github.com/GoogleCloudPlatform/DataflowJavaSDK)
>>   * Flink Dataflow runner (https://github.com/dataArtisans/flink-dataflow
>> )
>>   * Spark Dataflow runner (https://github.com/cloudera/spark-dataflow)
>>
>> These projects have always been Apache 2.0 licensed. We intend to bundle
>> all of these repositories since they are all complimentary and should be
>> maintained in one project. Prior to our submission, we will combine all
>> of these projects into a new git repository.
>>
>> == Source and Intellectual Property Submission Plan ==
>>
>> The source for the Dataflow SDK and the three runners (Spark, Flink,
>> Google Cloud Dataflow) are already licensed under an Apache 2 license.
>>
>>   * Dataflow SDK -
>> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENSE
>>   * Flink runner -
>> https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE
>>   * Spark runner -
>> https://github.com/cloudera/spark-dataflow/blob/master/LICENSE
>>
>> Contributors to the Dataflow SDK have also signed the Google Individual
>> Contributor License Agreement
>> (https://cla.developers.google.com/about/google-individual) in order to
>> contribute to the project.
>>
>> With respect to trademark rights, Google does not hold a trademark on
>> the phrase “Dataflow.” Based on feedback and guidance we receive during
>> the incubation process, we are open to renaming the project if necessary
>> for trademark or other concerns.
>>
>> == External Dependencies ==
>>
>> All external dependencies are licensed under an Apache 2.0 or
>> Apache-compatible license. As we grow the Beam community we will
>> configure our build process to require and validate all contributions
>> and dependencies are licensed under the Apache 2.0 license or are under
>> an Apache-compatible license.
>>
>> == Required Resources ==
>>
>> === Mailing Lists ===
>>
>> We currently use a mix of mailing lists. We will migrate our existing
>> mailing lists to the following:
>>
>>   * dev@beam.incubator.apache.org
>>   * user@beam.incubator.apache.org
>>   * private@beam.incubator.apache.org
>>   * commits@beam.incubator.apache.org
>>
>> === Source Control ===
>>
>> The Dataflow team currently uses Git and would like to continue to do
>> so. We request a Git repository for Beam with mirroring to GitHub enabled.
>>
>>   * https://git-wip-us.apache.org/repos/asf/incubator-beam.git
>>
>> === Issue Tracking ===
>>
>> We request the creation of an Apache-hosted JIRA. The Dataflow project
>> is currently using both a public GitHub issue tracker and internal
>> Google issue tracking. We will migrate and combine from these two
>> sources to the Apache JIRA.
>>
>>   * Jira ID: BEAM
>>
>> == Initial Committers ==
>>
>>   * Aljoscha Krettek        [aljoscha@apache.org]
>>   * Amit Sela               [amitsela33@gmail.com]
>>   * Ben Chambers            [bchambers@google.com]
>>   * Craig Chambers          [chambers@google.com]
>>   * Dan Halperin            [dhalperi@google.com]
>>   * Davor Bonaci            [davor@google.com]
>>   * Frances Perry           [fjp@google.com]
>>   * James Malone            [jamesmalone@google.com]
>>   * Jean-Baptiste Onofré    [jbonofre@apache.org]
>>   * Josh Wills              [jwills@apache.org]
>>   * Kostas Tzoumas          [kostas@data-artisans.com]
>>   * Kenneth Knowles         [klk@google.com]
>>   * Luke Cwik               [lcwik@google.com]
>>   * Maximilian Michels      [mxm@apache.org]
>>   * Stephan Ewen            [stephan@data-artisans.com]
>>   * Tom White               [tom@cloudera.com]
>>   * Tyler Akidau            [takidau@google.com]
>>   * Robert Bradshaw         [robertwb@google.com]
>>
>> == Additional Interested Contributors ==
>>
>>   * Debo Dutta              [dedutta@cisco.com]
>>   * Henry Saputra           [hsaputra@apache.org]
>>   * Taylor Goetz            [ptgoetz@gmail.com]
>>   * James Carman            [james@carmanconsulting.com]
>>   * Joe Witt                [joewitt@apache.org]
>>   * Vaibhav Gumashta        [vgumashta@hortonworks.com]
>>   * Prasanth Jayachandran   [pjayachandran@hortonworks.com]
>>   * Johan Edstrom           [seijoed@gmail.com]
>>   * Hugo Louro              [hmclouro@gmail.com]
>>   * Krzysztof Sobkowiak     [krzys.sobkowiak@gmail.com]
>>   * Jeff Genender           [jgenender@apache.org]
>>   * Edward J. Yoon          [edward.yoon@samsung.com]
>>   * Hao Chen                [hao@apache.org]
>>   * Byung-Gon Chun          [bgchun@gmail.com]
>>   * Charitha Elvitigala     [charithcc@apache.org]
>>   * Alexander Bezzubov      [bzz@apache.org]
>>   * Tsuyoshi Ozawa          [ozawa@apache.org]
>>   * Mayank Bansal           [mabansal@gmail.com]
>>   * Supun Kamburugamuve     [supun@apache.org]
>>   * Matthias Wessendorf     [matzew@apache.org]
>>   * Felix Cheung            [felixcheung@apache.org]
>>   * Ajay Yadava             [ajay.yadav@inmobi.com]
>>   * Liang Chen              [chenliang613@huawei.com]
>>   * Renaud Richardet        [renaud (at) apache (dot) org]
>>   * Bakey Pan               [bakey1985@gmail.com]
>>   * Andreas Neumann         [anew@apache.org]
>>   * Suresh Marru            [smarru@apache.org]
>>   * Hadrian Zbarcea         [hzbarcea@gmail.com]
>>
>> == Affiliations ==
>>
>> The initial committers are from six organizations. Google developed
>> Dataflow and the Dataflow SDK, data Artisans developed the Flink runner,
>> and Cloudera (Labs) developed the Spark runner.
>>
>>   * Cloudera
>>    * Tom White
>>   * Data Artisans
>>    * Aljoscha Krettek
>>    * Kostas Tzoumas
>>    * Maximilian Michels
>>    * Stephan Ewen
>>   * Google
>>    * Ben Chambers
>>    * Dan Halperin
>>    * Davor Bonaci
>>    * Frances Perry
>>    * James Malone
>>    * Kenneth Knowles
>>    * Luke Cwik
>>    * Tyler Akidau
>>    * Robert Bradshaw
>>   * PayPal
>>    * Amit Sela
>>   * Slack
>>    * Josh Wills
>>   * Talend
>>    * Jean-Baptiste Onofré
>>
>> == Sponsors ==
>>
>> === Champion ===
>>
>>   * Jean-Baptiste Onofre         [jbonofre@apache.org]
>>
>> === Nominated Mentors ===
>>
>>   * Jean-Baptiste Onofre       [jbonofre@apache.org]
>>   * Jim Jagielski              [jim@apache.org]
>>   * Venkatesh Seetharam        [venkatesh@apache.org]
>>   * Bertrand Delacretaz        [bdelacretaz@apache.org]
>>   * Ted Dunning                [tdunning@apache.org]
>>
>> === Sponsoring Entity ===
>>
>> The Apache Incubator
>> ----
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message