kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sophie Blee-Goldman (Jira)" <j...@apache.org>
Subject [jira] [Resolved] (KAFKA-3596) Kafka Streams: Window expiration needs to consider more than event time
Date Wed, 15 Jan 2020 22:20:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sophie Blee-Goldman resolved KAFKA-3596.
----------------------------------------
    Resolution: Not A Problem

> Kafka Streams: Window expiration needs to consider more than event time
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-3596
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3596
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>    Affects Versions: 0.10.0.0
>            Reporter: Henry Cai
>            Priority: Minor
>              Labels: architecture
>
> Currently in Kafka Streams, the way the windows are expired in RocksDB is triggered by
new event insertion.  When a window is created at T0 with 10 minutes retention, when we saw
a new record coming with event timestamp T0 + 10 +1, we will expire that window (remove it)
out of RocksDB.
> In the real world, it's very easy to see event coming with future timestamp (or out-of-order
events coming with big time gaps between events), this way of retiring a window based on one
event's event timestamp is dangerous.  I think at least we need to consider both the event's
event time and server/stream time elapse.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message