beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Groh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (BEAM-3337) Create an Executable Graph representation in the Universal Local Runner
Date Wed, 13 Dec 2017 00:58:00 GMT

     [ https://issues.apache.org/jira/browse/BEAM-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Groh updated BEAM-3337:
------------------------------
    Description: 
This represents the entirety of a Pipeline, divided into executable stages which are executed
in either the runner or within a user container. This representation must include the ports
over which the SDK harness communicates to the runner.

The construction of this graph likely includes most of the nodes present within the Runner
API graph (PTransform and PCollection), but injects additional nodes to represent a remote
read or write between harnesses.

Simple fusion (naive producer/consumer and sibling fusion) also should be performed here.

This will also require rewriting some boundary coders (e.g. runner -> SDK Harness and vice
versa) to be agnostic to the language of the runner harness, likely by converting into length-prefixed
bytes.

  was:
This represents the entirety of a Pipeline, divided into executable stages which are executed
in either the runner or within a user container. This representation must include the ports
over which the SDK harness communicates to the runner.

The construction of this graph likely includes most of the nodes present within the Runner
API graph (PTransform and PCollection), but injects additional nodes to represent a remote
read or write between harnesses.


> Create an Executable Graph representation in the Universal Local Runner
> -----------------------------------------------------------------------
>
>                 Key: BEAM-3337
>                 URL: https://issues.apache.org/jira/browse/BEAM-3337
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core
>            Reporter: Thomas Groh
>              Labels: portability
>
> This represents the entirety of a Pipeline, divided into executable stages which are
executed in either the runner or within a user container. This representation must include
the ports over which the SDK harness communicates to the runner.
> The construction of this graph likely includes most of the nodes present within the Runner
API graph (PTransform and PCollection), but injects additional nodes to represent a remote
read or write between harnesses.
> Simple fusion (naive producer/consumer and sibling fusion) also should be performed here.
> This will also require rewriting some boundary coders (e.g. runner -> SDK Harness
and vice versa) to be agnostic to the language of the runner harness, likely by converting
into length-prefixed bytes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message