beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Cwik (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2421) Migrate Dataflow to use impulse primitive as the only root primitive
Date Wed, 07 Jun 2017 21:40:18 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16041677#comment-16041677
] 

Luke Cwik commented on BEAM-2421:
---------------------------------

This is still in the prototyping phase so no tiny doc or discussion beyond what is occurring
here on this JIRA.

The byte array is one of the well known types across all SDKs. Its also very useful to make
it such that all IOs output/consume byte arrays and any formatting/conversion functions happen
within a ParDo/SplittableParDo allowing for a language to interpret that data in anyway it
chooses without needing to have a well known coder for the type that is shared across languages.

> Migrate Dataflow to use impulse primitive as the only root primitive
> --------------------------------------------------------------------
>
>                 Key: BEAM-2421
>                 URL: https://issues.apache.org/jira/browse/BEAM-2421
>             Project: Beam
>          Issue Type: Improvement
>          Components: beam-model-fn-api, beam-model-runner-api, runner-dataflow
>            Reporter: Luke Cwik
>
> The impulse source emits a single byte array element within the global window.
> This is in preparation for using SDF as the replacement for different bounded and unbounded
source implementations.
> Before:
> Read(Source) -> ParDo -> ...
> Impulse -> SDF(Source) -> ParDo -> ...



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message