flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudia Wegmann <c.wegm...@kasasi.de>
Subject AW: window-like use case
Date Fri, 23 Sep 2016 08:50:04 GMT

I'm no expert at all, but for me this sounds like a use case for Complex Event Processing
(CEP). I don't know if you're aware of Flinks CEP Library [1, 2]? Maybe that solves your problem
of multiple firings. But best to wait for the experts to answer your questions on handling
state and firing windows :)


[1]: https://flink.apache.org/news/2016/04/06/cep-monitoring.html
[2]: https://ci.apache.org/projects/flink/flink-docs-master/dev/libs/cep.html

-----Urspr√ľngliche Nachricht-----
Von: Maciek Próchniak [mailto:mpr@touk.pl] 
Gesendet: Freitag, 23. September 2016 10:36
An: user@flink.apache.org
Betreff: window-like use case


in our project we're dealing with a stream of billing events. Each has customerId and charge
amount We want to have a process that will trigger event (alarm) when sum of charges for customer
during last 4 hours exceeds certain threshold, say
- 10.
The triggered event should contain data from last billing event (the one that triggered alarm)

One one hand we can implement it as custom state - we'd save charges (or some precomputed
aggregates) from last 4 hours and trigger event when new one arrives.
OTOH we've been thinking if we can make it easier by using flink windows.
We tried to model our situation as sliding events (of length 4hours, slide is 1h), have some
precomputed aggregate and have a custom trigger that fires on element when threshold is exceeded.
It kinda works, except for the fact that state is unnecesarily large, custom trigger is a
bit hacky and (worst of all) when event with charge amount e.g. 20 arrives trigger fires in
all slides and we have duplicated events.
That's why we currently think about implementing it by custom state...

Do you have any other ideas/recommendations how can we handle such requirement?

View raw message