flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shaoxuan Wang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-6047) Master Jira for "Retraction for Flink Streaming"
Date Tue, 14 Mar 2017 15:50:41 GMT
Shaoxuan Wang created FLINK-6047:

             Summary: Master Jira for "Retraction for Flink Streaming"
                 Key: FLINK-6047
                 URL: https://issues.apache.org/jira/browse/FLINK-6047
             Project: Flink
          Issue Type: New Feature
            Reporter: Shaoxuan Wang
            Assignee: Shaoxuan Wang

[Design doc]:

"Retraction" is an important building block for data streaming to refine the early fired results
in streaming. “Early firing” are very common and widely used in many streaming scenarios,
for instance “window-less” or unbounded aggregate and stream-stream inner join, windowed
(with early firing) aggregate and stream-stream inner join. There are mainly two cases that
require retractions: 1) update on the keyed table (the key is either a primaryKey (PK) on
source table, or a groupKey/partitionKey in an aggregate); 2) When dynamic windows (e.g.,
session window) are in use, the new value may be replacing more than one previous window due
to window merging. 

To the best of our knowledge, the retraction for the early fired streaming results has never
been practically solved before. In this proposal, we develop a retraction solution and explain
how it works for the problem of “update on the keyed table”. The same solution can be
easily extended for the dynamic windows merging, as the key component of retraction - how
to refine an early fired results - is the same across different problems.  

[Proposed Jiras]:
Implement decoration phase for predicated logical plan rewriting after volcano optimization
Add source with table primary key and replace table property
Add sink tableInsert and NeedRetract property
Implement the retraction for partitioned unbounded over window aggregate
Implement the retraction for stream-stream inner join
Implement the retraction for the early firing window
Implement the retraction for the dynamic window with early firing

This message was sent by Atlassian JIRA

View raw message