beam-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Beam JIRA Bot (Jira)" <j...@apache.org>
Subject [jira] [Updated] (BEAM-7120) Turn `WindowedValue<T>` into `T` for all the interfaces of Data Plane
Date Wed, 10 Jun 2020 17:24:33 GMT

     [ https://issues.apache.org/jira/browse/BEAM-7120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Beam JIRA Bot updated BEAM-7120:
--------------------------------
    Labels:   (was: stale-assigned)

> Turn `WindowedValue<T>` into `T` for all the interfaces of Data Plane
> ---------------------------------------------------------------------
>
>                 Key: BEAM-7120
>                 URL: https://issues.apache.org/jira/browse/BEAM-7120
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-core, sdk-java-harness, sdk-py-harness
>            Reporter: sunjincheng
>            Priority: P2
>
> The main purpose of this JIRA is Turn `WindowedValue<T>` into `T` for all the interfaces
of Data Plane.
> The interface definitions for sending and receiving data in the code as follows:
>  - org.apache.beam.runners.fnexecution.data.FnDataService
> {code:java}
> public interface FnDataService {
>   <T> InboundDataClient receive(LogicalEndpoint inputLocation, Coder<WindowedValue<T>>
coder, FnDataReceiver<WindowedValue<T>> listener);
>   <T> CloseableFnDataReceiver<WindowedValue<T>> send(
>       LogicalEndpoint outputLocation, Coder<WindowedValue<T>> coder);
> }
> {code}
>  - org.apache.beam.fn.harness.data.BeamFnDataClient
> {code:java}
> public interface BeamFnDataClient {
>   <T> InboundDataClient receive(ApiServiceDescriptor apiServiceDescriptor, LogicalEndpoint
inputLocation, Coder<WindowedValue<T>> coder, FnDataReceiver<WindowedValue<T>>
receiver);
>   <T> CloseableFnDataReceiver<WindowedValue<T>> send(BeamFnDataGrpcClient
Endpoints.ApiServiceDescriptor apiServiceDescriptor, LogicalEndpoint outputLocation, Coder<WindowedValue<T>>
coder);
> }
> {code}
> Both `Coder<WindowedValue<T>>` and `FnDataReceiver<WindowedValue<T>>`
use `WindowedValue` as the data structure that both sides of Runner and SDK Harness know each
other. Control Plane/Data Plane/State Plane/Logging is a highly abstraction, such as Control
Plane and Logging, these are common requirements for all multi-language platforms. For example,
the Flink community is also discussing how to support Python UDF, as well as how to deal with
docker environment. how to data transfer, how to state access, how to logging etc. If Beam
can further abstract these service interfaces, i.e., interface definitions are compatible
with multiple engines, and finally provided to other projects in the form of class libraries,
it definitely will help other platforms that want to support multiple languages. So we can
further abstract the interface definition of FnDataService's BeamFnDataClient, take the FnDataService#receive
interface as an example, and turn `WindowedValue<T>` into `T` so that other platforms
can be extended arbitrarily, as follows:
> {code:java}
> <T> InboundDataClient receive(LogicalEndpoint inputLocation, Coder<T> coder,
FnDataReceiver<T>> listener);
> {code}
> For details, see the mail discussion list:
> [https://lists.apache.org/list.html?dev@beam.apache.org:lte=1M:%5BDISCUSS%5D%20Turn%20%60WindowedValue]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message