flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "William Saar" <will...@saar.se>
Subject Generating processing time watermarks in idle event time kafka streams?
Date Fri, 14 Dec 2018 08:52:39 GMT

Any standardized components to generate watermarks based on processing
time in an event time stream when there is no data from a source?

The docs for event time [1] indicate that people are doing this, but
the only suggestion on Stack Overflow [2] is to make every window
operator in stream have a special processing time trigger, which seems
clumsy and error prone if you have many jobs and many windows.

I'm thinking there might be a watermark generator that can be plugged
into the kafka consumer to make it generate processing time watermarks
when there are no messages? Or maybe a standardized operator that can
be inserted into the beginning of each stream, just after the source,
to generate such idle-source processing time watermarks in event time
streams?  

1. "Note that sometimes when event time programs are processing live
data in real-time, they will use some processing time operations in
order to guarantee that they are progressing in a timely fashion." 

2.
https://stackoverflow.com/questions/46432368/flink-window-does-not-process-data-at-end-of-stream




Mime
View raw message