flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Álvaro Vilaplana García <alvaro.vilapl...@gmail.com>
Subject Problem with ProcessFunction timeout feature
Date Thu, 22 Jun 2017 14:35:30 GMT

Please, can you help me with a problem? I summarise in the next points, I
hope is enough clear to approach some help.

a) We have devices, each with its own ID, which we don’t have control of

b) These devices send messages, with an internally generated, non-synced
(amongst other devices) timestamp

c) We want to detect when each devices may stop sending messages

d) For that, we are using a ProcessFunction

e) The devices put the messages in a Kafka topic, partitioned by ID.

f) We are struggling with the ProcessFunction timeout feature:

We cannot rely on real time (processing time), since the messages from the
devices may be delayed (even if their timestamp does not show these delays)
- so we rely on device timestamps instead.
In our case an event comes in which: "Registers a timer to be fired when
the event time watermark passes the given time". The problem we have is
there are cases where we do not get an additional event after the first
event- which means that the original event timeouts are not triggered.

As a side note we've seen in unit tests that flink seems to set a watermark
after the last event with a Long.MaxValue (9223372036854775807) - which
hides the above problem.

I am using Scala 2.11 /Flink versions 1.2.0


Álvaro Vilaplana García

View raw message