kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6556) allow to attach callbacks in kafka streams, to be triggered when a window is expired
Date Mon, 12 Mar 2018 11:33:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16395095#comment-16395095
] 

Abhishek Agarwal commented on KAFKA-6556:
-----------------------------------------

I am working on something similar (currently for tumbling windows). A construct to receive
only the final updates for a window is quite useful and powerful. 

> allow to attach callbacks in kafka streams, to be triggered when a window is expired

> -------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6556
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6556
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>            Reporter: igor mazor
>            Priority: Major
>
> Allowing to attach callbacks in kafka streams, to be triggered when a window is expired,
>  would help in use cases when the final state of the window is required.
>  It would be also useful if together with that functionally the user would be able to
control whether the callback would be triggered in addition to emitting the normal change
log down the stream, or only triggering the callback when the window is expired. (for example
in cases when only the final window state is required, and any updates to the window state
during the window time interval are not important)  
> An example for use case could be left join with proper sql semantics:
>  A left join B currently would emit (a1,null) and (a1,b1) if b1 arrives within the defined
join time window.
>  what I would like is to have ONLY ONE result:
>  (a1,null) if no b1 arrived during the defined join time window, OR ONLY (a1,b1) if b1
did arrived in the specified join time window.
>  One possible solution could be to use the current kafka streams left join with downstream
processor which would put the results in a windowed Ktable.
>  The window size would be same as for the left join operation, however, only the final
state of that window would be emitted down the stream once the time window is expired.
>  So if the left join produces (a1, null) and after X minutes no (a1, b1) was produced,
eventually only (a1, null) would be emitted, on the other hand, if the left join produces
(a1, null) and after X-t minutes (a1, b1) was produced by the left join operation => only
(a1, b1) would be emitted eventually down the stream after X minutes.
>  
> Another use case is when the window state is written to another kafka topic which is
then used to persist the window states into a db, However, many times only the final window
state
>  is required, and functionality to get only the last window state would help in reducing
load from the db, since only the final window state would be persisted to the db, instead
of multiple db writes for each window state update. 
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message