flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jaxbihani <jagad...@helpshift.com>
Subject Custom timer implementation using Flink
Date Tue, 11 Apr 2017 10:15:59 GMT
I have a use case which I am trying to solve using Flink. Need an advice to
decide on the correct approach.

Use case: 
 I have a stream of events partitioned by a key. For some events, I need to
start a timer (conside this as a SLA i.e. if something is not done in x
secs/minutes do something). Now when that timer expires I need to perform
some arbitrary action (like writing to database etc). There will be some
events which can cancel the timers. (i.e. if event comes before x secs we
need not run SLA violation action etc.). We are considering flink because
then we can reuse the scaling, fault tolerance provided by the engine rather
than building our own. Current rps is ~ 200-300 but it can be expected to
increase quickly.

Solutions in mind:
1. We can think it like CEP use case, where with encoding like "event1
followed by event2" with "not" in x seconds. i.e. when event 2 is "not"
arrived in x seconds. I assume there will be NOT operator support. I am not
sure about memory consumption in CEP. Because x seconds can be x days as
well and I do not need any batching of events in memory. I just need to
start a timer of x days/hours (lets say) and when it is fired just trigger
something. So there is no notion of window as such. Can CEP fit in this type
of use case? If the  timer between events is in days, how about the memory

2. Use Flink for event processing and delegate the tasks of timers to
another service i.e. when event occurs send it to kafka with timer
information and then another service handles timers and send back the event
again once that is done etc.  Looks like many hops in this process and
latency will be high if SLA is in seconds (I am thinking of using Kafka

Is anyone aware of a better way of doing this in flink? 

View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Custom-timer-implementation-using-Flink-tp12581.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

View raw message