flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Identifying missing events in keyed streams
Date Wed, 10 Oct 2018 13:54:09 GMT
Hi Averell,

I'd go with approach 2). As of Flink 1.6.0 you can delete timers.

But even if you are on a pre-1.6 version, a ProcessFunction would be the
way to go, IMO.
You don't need to register a timer for each event.
Instead, you can register the first timer with the first event and have a
state that is updated with the timestamp of the last seen event.
When the timer fires, you check the if you need to raise an alert and
register a new timer such that it fires 1 minute after the last seen event
(last-seen + 1 minute - (now - last-seen)).

Best, Fabian

Am Do., 4. Okt. 2018 um 16:15 Uhr schrieb Averell <lvhuyen@gmail.com>:

> Hi everyone,
>
> I have a keyed stream which is expecting events every fixed interval (let's
> say 1 minute). I want to raise alarms for any key which has received no
> events in n-periods. What should be the cheapest way (in term of
> performance
> ) to do this?
> I thought of some solutions, but don't know which one is the best:
> 1. Sliding window then count the number of events in each window <<< this
> seems quite expensive when n is big.
> 2. Register a timer for every single event, record the last event timestamp
> and check that timestamp when the timer expires. (This would be the best if
> there's an option to cancel/modify a timer, but it seems that feature is
> not
> available yet)
> 3. Session window: i haven't implemented this to verify its feasibility.
> Thinking of firing the alarm on every window clear event.
> 4. CEP. I don't know whether it's possible or not. Haven't found a guide
> for
> defining patterns of missing events.
>
> Could you please give some advices?
>
> Thanks and best regards,
> Averell
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>

Mime
View raw message