incubator-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marko Rodriguez <okramma...@gmail.com>
Subject Re: [DISCUSS] Apache Dataflow Incubator Proposal
Date Wed, 20 Jan 2016 16:44:38 GMT
Hi,

This is a cool idea. Its like Apache TinkerPop (http://tinkerpop.incubator.apache.org/) but
for data flow/stream systems as opposed to graph systems. Our tag line is "the JDBC for graphs."
You would be "the JDBC for streams." :)

You might be interested in looking TinkerPop's Gremlin language as its a data flow language.
	http://tinkerpop.apache.org/docs/3.1.0-incubating/#traversal
Moreover, its a virtual machine that allows other languages to compile to it.
	http://www.datastax.com/dev/blog/the-benefits-of-the-gremlin-graph-traversal-machine
		https://github.com/dkuppitz/sparql-gremlin
		https://github.com/twilmes/sql-gremlin

Do you have a URL to any documentation on Apache Dataflow's DSL? Perhaps there are ideas we
can steal!

Take care,
Marko.	

http://markorodriguez.com

On Jan 20, 2016, at 9:32 AM, James Malone <jamesmalone@google.com.INVALID> wrote:

> Hello everyone,
> 
> Attached to this message is a proposed new project - Apache Dataflow, a
> unified programming model for data processing and integration.
> 
> The text of the proposal is included below. Additionally, the proposal is
> in draft form on the wiki where we will make any required changes:
> 
> https://wiki.apache.org/incubator/DataflowProposal
> 
> We look forward to your feedback and input.
> 
> Best,
> 
> James
> 
> ----
> 
> = Apache Dataflow =
> 
> == Abstract ==
> 
> Dataflow is an open source, unified model and set of language-specific SDKs
> for defining and executing data processing workflows, and also data
> ingestion and integration flows, supporting Enterprise Integration Patterns
> (EIPs) and Domain Specific Languages (DSLs). Dataflow pipelines simplify
> the mechanics of large-scale batch and streaming data processing and can
> run on a number of runtimes like Apache Flink, Apache Spark, and Google
> Cloud Dataflow (a cloud service). Dataflow also brings DSL in different
> languages, allowing users to easily implement their data integration
> processes.
> 
> == Proposal ==
> 
> Dataflow is a simple, flexible, and powerful system for distributed data
> processing at any scale. Dataflow provides a unified programming model, a
> software development kit to define and construct data processing pipelines,
> and runners to execute Dataflow pipelines in several runtime engines, like
> Apache Spark, Apache Flink, or Google Cloud Dataflow. Dataflow can be used
> for a variety of streaming or batch data processing goals including ETL,
> stream analysis, and aggregate computation. The underlying programming
> model for Dataflow provides MapReduce-like parallelism, combined with
> support for powerful data windowing, and fine-grained correctness control.
> 
> == Background ==
> 
> Dataflow started as a set of Google projects focused on making data
> processing easier, faster, and less costly. The Dataflow model is a
> successor to MapReduce, FlumeJava, and Millwheel inside Google and is
> focused on providing a unified solution for batch and stream processing.
> These projects on which Dataflow is based have been published in several
> papers made available to the public:
> 
> * MapReduce - http://research.google.com/archive/mapreduce.html
> 
> * Dataflow model  - http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf
> 
> * FlumeJava - http://notes.stephenholiday.com/FlumeJava.pdf
> 
> * MillWheel - http://research.google.com/pubs/pub41378.html
> 
> Dataflow was designed from the start to provide a portable programming
> layer. When you define a data processing pipeline with the Dataflow model,
> you are creating a job which is capable of being processed by any number of
> Dataflow processing engines. Several engines have been developed to run
> Dataflow pipelines in other open source runtimes, including a Dataflow
> runner for Apache Flink and Apache Spark. There is also a “direct runner”,
> for execution on the developer machine (mainly for dev/debug purposes).
> Another runner allows a Dataflow program to run on a managed service,
> Google Cloud Dataflow, in Google Cloud Platform. The Dataflow Java SDK is
> already available on GitHub, and independent from the Google Cloud Dataflow
> service. Another Python SDK is currently in active development.
> 
> In this proposal, the Dataflow SDKs, model, and a set of runners will be
> submitted as an OSS project under the ASF. The runners which are a part of
> this proposal include those for Spark (from Cloudera), Flink (from data
> Artisans), and local development (from Google); the Google Cloud Dataflow
> service runner is not included in this proposal. Further references to
> Dataflow will refer to the Dataflow model, SDKs, and runners which are a
> part of this proposal (Apache Dataflow) only. The initial submission will
> contain the already-released Java SDK; Google intends to submit the Python
> SDK later in the incubation process. The Google Cloud Dataflow service will
> continue to be one of many runners for Dataflow, built on Google Cloud
> Platform, to run Dataflow pipelines. Necessarily, Cloud Dataflow will
> develop against the Apache project additions, updates, and changes. Google
> Cloud Dataflow will become one user of Apache Dataflow and will participate
> in the project openly and publicly.
> 
> The Dataflow programming model has been designed with simplicity,
> scalability, and speed as key tenants. In the Dataflow model, you only need
> to think about four top-level concepts when constructing your data
> processing job:
> 
> * Pipelines - The data processing job made of a series of computations
> including input, processing, and output
> 
> * PCollections - Bounded (or unbounded) datasets which represent the input,
> intermediate and output data in pipelines
> 
> * PTransforms - A data processing step in a pipeline in which one or more
> PCollections are an input and output
> 
> * I/O Sources and Sinks - APIs for reading and writing data which are the
> roots and endpoints of the pipeline
> 
> == Rationale ==
> 
> With Dataflow, Google intended to develop a framework which allowed
> developers to be maximally productive in defining the processing, and then
> be able to execute the program at various levels of
> latency/cost/completeness without re-architecting or re-writing it. This
> goal was informed by Google’s past experience  developing several models,
> frameworks, and tools useful for large-scale and distributed data
> processing. While Google has previously published papers describing some of
> its technologies, Google decided to take a different approach with
> Dataflow. Google open-sourced the SDK and model alongside commercialization
> of the idea and ahead of publishing papers on the topic. As a result, a
> number of open source runtimes exist for Dataflow, such as the Apache Flink
> and Apache Spark runners.
> 
> We believe that submitting Dataflow as an Apache project will provide an
> immediate, worthwhile, and substantial contribution to the open source
> community. As an incubating project, we believe Dataflow will have a better
> opportunity to provide a meaningful contribution to OSS and also integrate
> with other Apache projects.
> 
> In the long term, we believe Dataflow can be a powerful abstraction layer
> for data processing. By providing an abstraction layer for data pipelines
> and processing, data workflows can be increasingly portable, resilient to
> breaking changes in tooling, and compatible across many execution engines,
> runtimes, and open source projects.
> 
> == Initial Goals ==
> 
> We are breaking our initial goals into immediate (< 2 months), short-term
> (2-4 months), and intermediate-term (> 4 months).
> 
> Our immediate goals include the following:
> 
> * Plan for reconciling the Dataflow Java SDK and various runners into one
> project
> 
> * Plan for refactoring the existing Java SDK for better extensibility by
> SDK and runner writers
> 
> * Validating all dependencies are ASL 2.0 or compatible
> 
> * Understanding and adapting to the Apache development process
> 
> Our short-term goals include:
> 
> * Moving the newly-merged lists, and build utilities to Apache
> 
> * Start refactoring codebase and move code to Apache Git repo
> 
> * Continue development of new features, functions, and fixes in the
> Dataflow Java SDK, and Dataflow runners
> 
> * Cleaning up the Dataflow SDK sources and crafting a roadmap and plan for
> how to include new major ideas, modules, and runtimes
> 
> * Establishment of easy and clear build/test framework for Dataflow and
> associated runtimes; creation of testing, rollback, and validation policy
> 
> * Analysis and design for work needed to make Dataflow a better data
> processing abstraction layer for multiple open source frameworks and
> environments
> 
> Finally, we have a number of intermediate-term goals:
> 
> * Roadmapping, planning, and execution of integrations with other OSS and
> non-OSS projects/products
> 
> * Inclusion of additional SDK for Python, which is under active development
> 
> == Current Status ==
> 
> === Meritocracy ===
> 
> Dataflow was initially developed based on ideas from many employees within
> Google. As an ASL OSS project on GitHub, the Dataflow SDK has received
> contributions from data Artisans, Cloudera Labs, and other individual
> developers. As a project under incubation, we are committed to expanding
> our effort to build an environment which supports a meritocracy. We are
> focused on engaging the community and other related projects for support
> and contributions. Moreover, we are committed to ensure contributors and
> committers to Dataflow come from a broad mix of organizations through a
> merit-based decision process during incubation. We believe strongly in the
> Dataflow model and are committed to growing an inclusive community of
> Dataflow contributors.
> 
> === Community ===
> 
> The core of the Dataflow Java SDK has been developed by Google for use with
> Google Cloud Dataflow. Google has active community engagement in the SDK
> GitHub repository (https://github.com/GoogleCloudPlatform/DataflowJavaSDK),
> on Stack Overflow (
> http://stackoverflow.com/questions/tagged/google-cloud-dataflow) and has
> had contributions from a number of organizations and indivuduals.
> 
> Everyday, Cloud Dataflow is actively used by a number of organizations and
> institutions for batch and stream processing of data. We believe acceptance
> will allow us to consolidate existing Dataflow-related work, grow the
> Dataflow community, and deepen connections between Dataflow and other open
> source projects.
> 
> === Core Developers ===
> 
> The core developers for Dataflow and the Dataflow runners are:
> 
> * Frances Perry
> 
> * Tyler Akidau
> 
> * Davor Bonaci
> 
> * Luke Cwik
> 
> * Ben Chambers
> 
> * Kenn Knowles
> 
> * Dan Halperin
> 
> * Daniel Mills
> 
> * Mark Shields
> 
> * Craig Chambers
> 
> * Maximilian Michels
> 
> * Tom White
> 
> * Josh Wills
> 
> === Alignment ===
> 
> The Dataflow SDK can be used to create Dataflow pipelines which can be
> executed on Apache Spark or Apache Flink. Dataflow is also related to other
> Apache projects, such as Apache Crunch. We plan on expanding functionality
> for Dataflow runners, support for additional domain specific languages, and
> increased portability so Dataflow is a powerful abstraction layer for data
> processing.
> 
> == Known Risks ==
> 
> === Orphaned Products ===
> 
> The Dataflow SDK is presently used by several organizations, from small
> startups to Fortune 100 companies, to construct production pipelines which
> are executed in Google Cloud Dataflow. Google has a long-term commitment to
> advance the Dataflow SDK; moreover, Dataflow is seeing increasing interest,
> development, and adoption from organizations outside of Google.
> 
> === Inexperience with Open Source ===
> 
> Google believes strongly in open source and the exchange of information to
> advance new ideas and work. Examples of this commitment are active OSS
> projects such as Chromium (https://www.chromium.org) and Kubernetes (
> http://kubernetes.io/). With Dataflow, we have tried to be increasingly
> open and forward-looking; we have published a paper in the VLDB conference
> describing the Dataflow model (
> http://www.vldb.org/pvldb/vol8/p1792-Akidau.pdf) and were quick to release
> the Dataflow SDK as open source software with the launch of Cloud Dataflow.
> Our submission to the Apache Software Foundation is a logical extension of
> our commitment to open source software.
> 
> === Homogeneous Developers ===
> 
> The majority of committers in this proposal belong to Google due to the
> fact that Dataflow has emerged from several internal Google projects. This
> proposal also includes committers outside of Google who are actively
> involved with other Apache projects, such as Hadoop, Flink, and Spark.  We
> expect our entry into incubation will allow us to expand the number of
> individuals and organizations participating in Dataflow development.
> Additionally, separation of the Dataflow SDK from Google Cloud Dataflow
> allows us to focus on the open source SDK and model and do what is best for
> this project.
> 
> === Reliance on Salaried Developers ===
> 
> The Dataflow SDK and Dataflow runners have been developed primarily by
> salaried developers supporting the Google Cloud Dataflow project. While the
> Dataflow SDK and Cloud Dataflow have been developed by different teams (and
> this proposal would reinforce that separation) we expect our initial set of
> developers will still primarily be salaried. Contribution has not been
> exclusively from salaried developers, however. For example, the contrib
> directory of the Dataflow SDK (
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/tree/master/contrib)
> contains items from free-time contributors. Moreover, seperate projects,
> such as ScalaFlow (https://github.com/darkjh/scalaflow) have been created
> around the Dataflow model and SDK. We expect our reliance on salaried
> developers will decrease over time during incubation.
> 
> === Relationship with other Apache products ===
> 
> Dataflow directly interoperates with or utilizes several existing Apache
> projects.
> 
> * Build
> 
> ** Apache Maven
> 
> * Data I/O, Libraries
> 
> ** Apache Avro
> 
> ** Apache Commons
> 
> * Dataflow runners
> 
> ** Apache Flink
> 
> ** Apache Spark
> 
> Dataflow when used in batch mode shares similarities with Apache Crunch;
> however, Dataflow is focused on a model, SDK, and abstraction layer beyond
> Spark and Hadoop (MapReduce.) One key goal of Dataflow is to provide an
> intermediate abstraction layer which can easily be implemented and utilized
> across several different processing frameworks.
> 
> === An excessive fascination with the Apache brand ===
> 
> With this proposal we are not seeking attention or publicity. Rather, we
> firmly believe in the Dataflow model, SDK, and the ability to make Dataflow
> a powerful yet simple framework for data processing. While the Dataflow SDK
> and model have been open source, we believe putting code on GitHub can only
> go so far. We see the Apache community, processes, and mission as critical
> for ensuring the Dataflow SDK and model are truly community-driven,
> positively impactful, and innovative open source software. While Google has
> taken a number of steps to advance its various open source projects, we
> believe Dataflow is a great fit for the Apache Software Foundation due to
> its focus on data processing and its relationships to existing ASF projects.
> 
> == Documentation ==
> 
> The following documentation is relevant to this proposal. Relevant portion
> of the documentation will be contributed to the Apache Dataflow project.
> 
> * Dataflow website: https://cloud.google.com/dataflow
> 
> * Dataflow programming model:
> https://cloud.google.com/dataflow/model/programming-model
> 
> * Codebases
> 
> ** Dataflow Java SDK: https://github.com/GoogleCloudPlatform/DataflowJavaSDK
> 
> ** Flink Dataflow runner: https://github.com/dataArtisans/flink-dataflow
> 
> ** Spark Dataflow runner: https://github.com/cloudera/spark-dataflow
> 
> * Dataflow Java SDK issue tracker:
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/issues
> 
> * google-cloud-dataflow tag on Stack Overflow:
> http://stackoverflow.com/questions/tagged/google-cloud-dataflow
> 
> == Initial Source ==
> 
> The initial source for Dataflow which we will submit to the Apache
> Foundation will include several related projects which are currently hosted
> on the GitHub repositories:
> 
> * Dataflow Java SDK (https://github.com/GoogleCloudPlatform/DataflowJavaSDK)
> 
> * Flink Dataflow runner (https://github.com/dataArtisans/flink-dataflow)
> 
> * Spark Dataflow runner (https://github.com/cloudera/spark-dataflow)
> 
> These projects have always been Apache 2.0 licensed. We intend to bundle
> all of these repositories since they are all complimentary and should be
> maintained in one project. Prior to our submission, we will combine all of
> these projects into a new git repository.
> 
> == Source and Intellectual Property Submission Plan ==
> 
> The source for the Dataflow SDK and the three runners (Spark, Flink, Google
> Cloud Dataflow) are already licensed under an Apache 2 license.
> 
> * Dataflow SDK -
> https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/LICENSE
> 
> * Flink runner -
> https://github.com/dataArtisans/flink-dataflow/blob/master/LICENSE
> 
> * Spark runner -
> https://github.com/cloudera/spark-dataflow/blob/master/LICENSE
> 
> Contributors to the Dataflow SDK have also signed the Google Individual
> Contributor License Agreement (
> https://cla.developers.google.com/about/google-individual) in order to
> contribute to the project.
> 
> With respect to trademark rights, Google does not hold a trademark on the
> phrase “Dataflow.” Based on feedback and guidance we receive during the
> incubation process, we are open to renaming the project if necessary for
> trademark or other concerns.
> 
> == External Dependencies ==
> 
> All external dependencies are licensed under an Apache 2.0 or
> Apache-compatible license. As we grow the Dataflow community we will
> configure our build process to require and validate all contributions and
> dependencies are licensed under the Apache 2.0 license or are under an
> Apache-compatible license.
> 
> == Required Resources ==
> 
> === Mailing Lists ===
> 
> We currently use a mix of mailing lists. We will migrate our existing
> mailing lists to the following:
> 
> * dev@dataflow.incubator.apache.org
> 
> * user@dataflow.incubator.apache.org
> 
> * private@dataflow.incubator.apache.org
> 
> * commits@dataflow.incubator.apache.org
> 
> === Source Control ===
> 
> The Dataflow team currently uses Git and would like to continue to do so.
> We request a Git repository for Dataflow with mirroring to GitHub enabled.
> 
> === Issue Tracking ===
> 
> We request the creation of an Apache-hosted JIRA. The Dataflow project is
> currently using both a public GitHub issue tracker and internal Google
> issue tracking. We will migrate and combine from these two sources to the
> Apache JIRA.
> 
> == Initial Committers ==
> 
> * Aljoscha Krettek     [aljoscha@apache.org]
> 
> * Amit Sela            [amitsela33@gmail.com]
> 
> * Ben Chambers         [bchambers@google.com]
> 
> * Craig Chambers       [chambers@google.com]
> 
> * Dan Halperin         [dhalperi@google.com]
> 
> * Davor Bonaci         [davor@google.com]
> 
> * Frances Perry        [fjp@google.com]
> 
> * James Malone         [jamesmalone@google.com]
> 
> * Jean-Baptiste Onofré [jbonofre@apache.org]
> 
> * Josh Wills           [jwills@apache.org]
> 
> * Kostas Tzoumas       [kostas@data-artisans.com]
> 
> * Kenneth Knowles      [klk@google.com]
> 
> * Luke Cwik            [lcwik@google.com]
> 
> * Maximilian Michels   [mxm@apache.org]
> 
> * Stephan Ewen         [stephan@data-artisans.com]
> 
> * Tom White            [tom@cloudera.com]
> 
> * Tyler Akidau         [takidau@google.com]
> 
> == Affiliations ==
> 
> The initial committers are from six organizations. Google developed
> Dataflow and the Dataflow SDK, data Artisans developed the Flink runner,
> and Cloudera (Labs) developed the Spark runner.
> 
> * Cloudera
> 
> ** Tom White
> 
> * Data Artisans
> 
> ** Aljoscha Krettek
> 
> ** Kostas Tzoumas
> 
> ** Maximilian Michels
> 
> ** Stephan Ewen
> 
> * Google
> 
> ** Ben Chambers
> 
> ** Dan Halperin
> 
> ** Davor Bonaci
> 
> ** Frances Perry
> 
> ** James Malone
> 
> ** Kenneth Knowles
> 
> ** Luke Cwik
> 
> ** Tyler Akidau
> 
> * PayPal
> 
> ** Amit Sela
> 
> * Slack
> 
> ** Josh Wills
> 
> * Talend
> 
> ** Jean-Baptiste Onofré
> 
> == Sponsors ==
> 
> === Champion ===
> 
> * Jean-Baptiste Onofre      [jbonofre@apache.org]
> 
> === Nominated Mentors ===
> 
> * Jim Jagielski           [jim@apache.org]
> 
> * Venkatesh Seetharam     [venkatesh@apache.org]
> 
> * Bertrand Delacretaz     [bdelacretaz@apache.org]
> 
> * Ted Dunning             [tdunning@apache.org]
> 
> === Sponsoring Entity ===
> 
> The Apache Incubator


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message