flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "William Saar" <will...@saar.se>
Subject Re: ContinuousEventTimeTrigger breaks coGrouped windowed streams?
Date Mon, 21 Nov 2016 17:52:31 GMT

Yes, the SlidingEventTimeWindow works, but is there any way to
pre-aggregate things with tumbling windows that emit events more often
than their window size? Perhaps I can do this before I merge the
streams? (But if ContinuousEventTimeTrigger is the only way to do
that, it's bad if it doesn't clean up its state).

I assume using sliding window states will be too large and less
efficient than tumbling windows as a sliding fold needs to keep all
events in the window and recompute the fold as events slide out of the
window, while a tumbling fold just needs to keep the aggregation and
can discard events as it folds them. 

I am reviewing how one would replace a batch solution based on 3
bucketed aggregations of different window sizes and it seems tumbling
windows would be a perfect fit and would need to keep only the 3
aggregations a memory, while sliding windows would need to keep up to
3 copies of all events (for at least the smallest window size) to
compute the same type of results.


----- Original Message -----
From: user@flink.apache.org
Sent:Mon, 21 Nov 2016 08:22:16 +0000
Subject:Re: ContinuousEventTimeTrigger breaks coGrouped windowed

Hi William,

I am wondering whether the ContinuousEventTimeTrigger is the best
choice here (it never cleans up the state as far as I know).

Have you tried the simple SlidingEventTimeWindows as your window

William Saar <william@saar.se [1]> ezt írta (időpont: 2016. nov.
19., Szo, 18:28):

	My topology below seems to work when I comment out all the lines with
ContinuousEventTimeTrigger, but prints nothing when the line is in
there. Can I coGroup two large time windows that use a different
trigger time than the window size? (even if the
ContinuousEventTimeTrigger doesn't work for coGroups, I would not
expect the result to be completely silent). 

	The streams I'm cogroupng are from 2 different Kafka sources and uses
event time with 0 out of orderness and I'm on Flink 1.1.3, if that

	DataStream<CommonType> stream1 =
     <stream of event type1>

	DataStream<CommonType> stream2 =
     <stream of event type2>



	Thanks, William


[1] mailto:william@saar.se

View raw message