flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Elias Levy <fearsome.lucid...@gmail.com>
Subject Re: Event processing time with lateness
Date Mon, 06 Jun 2016 23:53:29 GMT
On Fri, Jun 3, 2016 at 6:47 AM, Kostas Kloudas <k.kloudas@data-artisans.com>

> To see a relatively more complex example of such a trigger and how to
> implement it,
> you can have a look at this implementation:
> https://github.com/dataArtisans/beam_comp/blob/master/src/main/java/com/dataartisans/beam_comparison/customTriggers/EventTimeTriggerWithEarlyAndLateFiring.java

I've modified this trigger so that firing are suppressed unless there are
new events between timers.  This can significantly reduce the outputted
events, which could mean much reduced writes to a downstream data store.
See https://gist.github.com/eliaslevy/ec840444607b9a5dd5aa3eb2cdd77932.

Also, I find the accumulating behavior somewhat unintuitive as when
disabled it only purges when the time window ends.  When discarding is in
effect, it seems more natural for purging it to occur at each firing,
whether early, at the windows event time end, or late.  Otherwise, you may
end up with output events of different semantics.  E.g. with the current
behavior if you are implementing a counter early firing will result on
partial counts until the window end, after that late will give you partial
counts of the delta from the window end count.  It would be more consistent
to either generate partial counts at all firing or deltas at all firing, so
that the output of the operator can be processes the same downstream.

View raw message