flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Any good ideas for online/offline detection of devices that send events?
Date Fri, 10 Mar 2017 19:34:52 GMT
pro tip for debugging watermarks: They are exposed via a metric in Flink

On Tue, Mar 7, 2017 at 1:37 PM, Bruno Aranda <brunoaranda@gmail.com> wrote:

> Hi Gordon,
> Many thanks for your helpful ideas. We tried yesterday the CEP approach,
> but could not figure it out. The ProcessFunction one looks more promising,
> and we are investigating it, though we are fighting with some issues
> related to the event time, where we cannot see so far the timer triggered
> at the right event time. We are using ascending timestamps, but at the
> moment we see the timers fired when it is too late. Investigating more.
> Thanks,
> Bruno
> On Tue, 7 Mar 2017 at 07:49 Tzu-Li (Gordon) Tai <tzulitai@apache.org>
> wrote:
>> Some more input:
>> Right now, you can also use the `ProcessFunction` [1] available in Flink
>> 1.2 to simulate state TTL.
>> The `ProcessFunction` should allow you to keep device state and simulate
>> the online / offline detection by registering processing timers. In the
>> `onTimer` callback, you can emit the “offline” marker event downstream, and
>> in the `processElement` method, you can emit the “online” marker event if
>> the case is the device has sent an event after it was determined to be
>> offline.
>> [1] https://ci.apache.org/projects/flink/flink-docs-
>> release-1.2/dev/stream/process_function.html
>> On March 6, 2017 at 9:40:28 PM, Tzu-Li (Gordon) Tai (tzulitai@apache.org)
>> wrote:
>> Hi Bruno!
>> The Flink CEP library also seems like an option you can look into to see
>> if it can easily realize what you have in mind.
>> Basically, the pattern you are detecting is a timeout of 5 minutes after
>> the last event. Once that pattern is detected, you emit a “device offline”
>> event downstream.
>> With this, you can also extend the pattern output stream to detect
>> whether a device has became online again.
>> Here are some materials for you to take a look at Flink CEP:
>> 1. http://flink.apache.org/news/2016/04/06/cep-monitoring.html
>> 2. https://www.slideshare.net/FlinkForward/fabian-huesketill-rohrmann-
>> declarative-stream-processing-with-streamsql-and-cep?qid=
>> 3c13eb7d-ed39-4eae-9b74-a6c94e8b08a3&v=&b=&from_search=4
>> The CEP parts in the slides in 2. also provides some good examples of
>> timeout detection using CEP.
>> Hope this helps!
>> Cheers,
>> Gordon
>> On March 4, 2017 at 1:27:51 AM, Bruno Aranda (baranda@apache.org) wrote:
>> Hi all,
>> We are trying to write an online/offline detector for devices that keep
>> streaming data through Flink. We know how often roughly to expect events
>> from those devices and we want to be able to detect when any of them stops
>> (goes offline) or starts again (comes back online) sending events through
>> the pipeline. For instance, if 5 minutes have passed since the last event
>> of a device, we would fire an event to indicate that the device is offline.
>> The data from the devices comes through Kafka, with their own event time.
>> The devices events are in order in the partitions and each devices goes to
>> a specific partition, so in theory, we should not have out of order when
>> looking at one partition.
>> We are assuming a good way to do this is by using sliding windows that
>> are big enough, so we can see the relevant gap before/after the events for
>> each specific device.
>> We were wondering if there are other ideas on how to solve this.
>> Many thanks!
>> Bruno

View raw message