beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ryan Brush (JIRA)" <>
Subject [jira] [Commented] (BEAM-11) Integrate Spark runner with Beam
Date Mon, 07 Mar 2016 17:27:40 GMT


Ryan Brush commented on BEAM-11:

Not sure if others are working on this, but the commit linked below is probably the smallest
possible change to get spark-dataflow running with the current Beam code. 

Here it is:

We need to do quite a bit more with the runner before it's broadly usable; see the ugly "TODO"
around state internals in the commit. So perhaps the best path forward is to just create a
development branch of Beam that includes the dataflow runner and we can improve on it there?
Once it's in a better state we can squash/rebase (or whatever conventions this project follows)
to get a clean merge into master.

I'm happy to create the branch if desired (although I lack commit privs), or feel free to
just grab the code from the above commit if it makes sense.

> Integrate Spark runner with Beam
> --------------------------------
>                 Key: BEAM-11
>                 URL:
>             Project: Beam
>          Issue Type: Task
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Amit Sela
> Refactor and integrate the Spark runner code against Google's contributed version of
Dataflow - Beam.

This message was sent by Atlassian JIRA

View raw message