flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elias Levy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-6239) Sharing of State Across Operators
Date Thu, 05 Jul 2018 00:35:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16533142#comment-16533142
] 

Elias Levy commented on FLINK-6239:
-----------------------------------

For our particular use case having a single operator that is owner and can write, and other
that can read the state, is sufficient.  The idea was essentially to implement a materialized
table join (similar to Kafka Streams KTable).  We have need to join again the table multiple
times, but the table (state) would be of significant size, and duplicating it is something
we rather not do.

That said, I think we may be able to get around this issue by creating a generic wrapping
container, unioning all the streams, and feeding it to a single operator that perform all
the work, instead of having multiple operators and multiple streams.  Not as clean, but probably
workable.

> Sharing of State Across Operators
> ---------------------------------
>
>                 Key: FLINK-6239
>                 URL: https://issues.apache.org/jira/browse/FLINK-6239
>             Project: Flink
>          Issue Type: Improvement
>          Components: DataStream API
>    Affects Versions: 1.1.4
>            Reporter: Elias Levy
>            Priority: Major
>
> Currently state cannot be shared across operators.  On a keyed stream, the state is implicitly
keyed by the operator id, in addition to the stream key.  This can make it more difficult
and inefficient to implement complex topologies, where multiple operator may need to access
the same state.  It would be value to be able to access keyed value and map stated across
operators.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message