flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sameer W <sam...@axiomine.com>
Subject CEP and slightly out of order elements
Date Tue, 11 Oct 2016 11:51:38 GMT
Hi,

If using CEP with event-time I have events which can be slightly out of
order and I want to sort them by timestamp within their time-windows before
applying CEP-

For example, if using 5 second windows and I use the following

ds2 = ds.keyBy.window(TumblingWindow(10 seconds).apply(/*Sort by
Timestamp*/);

Next assign watermarks again on ds2 (because elements in ds2 will all have
the same timestamp of WINDOW_END_TIME-1ms)
ds2.assignTimestampsAndWatermarks()

Finally apply CEP on ds2 with a WITHIN window of 5 seconds (shorter
timestamp than the one I used earlier).

The reasoning is, if I am using the next() operator in CEP, the events
should be in the order of their timestamps.

Is this the right way to handle this problem? I have heard people say that
assigning watermarks twice can lead to wrong results. But don't I need to
assign timestamps once more in this scenario.

Thanks,
Sameer

Mime
View raw message