beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-11) Integrate Spark runner with Beam
Date Mon, 07 Mar 2016 18:10:40 GMT

    [ https://issues.apache.org/jira/browse/BEAM-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183372#comment-15183372
] 

Amit Sela commented on BEAM-11:
-------------------------------

Yes, I'm on it :) pending the code drop. Hang tight..
I'll take a look at your work, and compare with mine. I'd merge our work, or tell you to do
a pull request, but this is a weird time since there is still no code... But no worries, the
code will be dropped soon and we can get things going.

There is a lot of work to do on the Beam model, especially in streaming, but also in metric
reporting and more, as stated in the technical vision document: https://drive.google.com/folderview?id=0B-IhJZh9Ab52OFBVZHpsNjc4eXc&usp=sharing

Currently, the runner supports batch processing, and some, limited, stream processing, which
is OK - we can state the runner's current capabilities (keep in mind that not all runners
will support the entire model). Having said that, we will add more support as we go.

Concerning branch organization see: https://drive.google.com/folderview?id=0B-IhJZh9Ab52OFBVZHpsNjc4eXc&usp=sharing
It makes total sense to develop large features of the runner in a feature branch, and merge
once done.

> Integrate Spark runner with Beam
> --------------------------------
>
>                 Key: BEAM-11
>                 URL: https://issues.apache.org/jira/browse/BEAM-11
>             Project: Beam
>          Issue Type: Task
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Amit Sela
>
> Refactor and integrate the Spark runner code against Google's contributed version of
Dataflow - Beam.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message