flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2393) Add a "stateless at-least-once" mode for streaming
Date Wed, 22 Jul 2015 15:24:04 GMT
Stephan Ewen created FLINK-2393:

             Summary: Add a "stateless at-least-once" mode for streaming
                 Key: FLINK-2393
                 URL: https://issues.apache.org/jira/browse/FLINK-2393
             Project: Flink
          Issue Type: New Feature
          Components: Streaming
    Affects Versions: 0.10
            Reporter: Stephan Ewen

Currently, the checkpointing mechanism provides "exactly once" guarantees. Part of that is
the step that temporarily "aligns" the data streams. This step increases the tuple latency

By offering a version that does not provide "exactly-once", but only "at-least-once", we can
avoid the latency increase. For super-low-latency applications, that tolerate duplicates,
this may be an interesting option.

To realize that, we would use a slightly modified version of the checkpointing algorithm.
Effectively, the streams would not be aligned, but tasks would only count the received barriers
and emit their own barrier as soon as the saw a barrier from all inputs.

My feeling is that it makes not sense to implement state backups, when being concerned with
this super low latency. The mode would hence be a purely stateless at-least-once mode.

This message was sent by Atlassian JIRA

View raw message