flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elias Levy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-4558) Add support for synchronizing streams
Date Thu, 01 Sep 2016 16:24:20 GMT
Elias Levy created FLINK-4558:

             Summary: Add support for synchronizing streams
                 Key: FLINK-4558
                 URL: https://issues.apache.org/jira/browse/FLINK-4558
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
    Affects Versions: 1.1.0
            Reporter: Elias Levy

As mentioned on the [mailing list|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/synchronizing-two-streams-td6830.html],
there are use cases that require synchronizing two streams on via their times and where it
is not practical to buffer all messages from one streams while waiting for the other to synchronize.
 Flink should add functionality to enable such use cases.

This could be implemented by modifying TwoInputStreamOperator so that calls to processElement1
and processElement2 could return a value indicating that the element can't yet be processed,
having the framework then pause processing for some time, potentially using exponential back
off with a hard maximum, and then allowing the back pressure system to do its work and pause
the stream.

Alternatively, an API could be added to explicitly pause/unpause a stream.

For ease of use either of these mechanism should be used to create a SynchronizedTwoInputStreamOperator
that end users can utilize by passing a configurable time delta to use as a synchronization

This message was sent by Atlassian JIRA

View raw message