flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Timer coalescing necessary?
Date Fri, 13 Oct 2017 12:48:37 GMT
Hi,

This is slightly different for processing-time and event-time triggers.

First, event-time triggers: there are two data structures, a PriorityQueue (which is implemented
as a heap) of timers that is sorted by timestamp, a set of registered timers that is used
for deduplication. When adding a timer, we first check whether it already exists (using the
set) and then add it to the queue. Whenever we receive a watermark we poll from the timer
queue as long as the timestamp of the top timer is <= the watermark. We remote the timer
from the set and call the user callback.

For processing-time triggers it's very similar, except that we use a ProcessingTimeService
instead of the watermark for advancing time. We always have one "physical" processing-time
timer set at the ProcessingTimeService. When this fires we follow the same procedure as for
event-time and then register a new "physical" timer for the next lowest processing-time timer.

In you case this would mean 3 separate internal timers, but a timer is only a timestamp and
a key (and a namespace). 

Best,
Aljoscha


> On 13. Oct 2017, at 13:56, Kien Truong <duckientruong@gmail.com> wrote:
> 
> Hi Aljoscha,
> 
> Could you clarify how the timer system works right now ?
> 
> For example, let's say I have a function F, with 3 keys that are registered to execute
at processing time T. 
> Would Flink maintain a single internal timer at time T, then run the callback on all
3 keys when it's triggered ? Or there'd be 3 internal timers that will be triggered separately
at time T  ?
> 
> Best regards,
> 
> Kien
> On 10/13/2017 6:43 PM, Aljoscha Krettek wrote:
>> Hi,
>> 
>> If you have multiple timers per key, then coalescing can make sense to reduce the
burden on the timer system. Coalescing them across different keys would not be possible right
now.
>> 
>> Best,
>> Aljoscha
>> 
>> 
>>> On 13. Oct 2017, at 06:37, Kien Truong <duckientruong@gmail.com>
>>>  wrote:
>>> 
>>> Hi,
>>> 
>>> We are having a streaming job where we use timers to implement key timeout for
stateful functions. Should we implement coalescing logic to reduce the number of timer trigger,
or it is not necessary with Flink?
>>> 
>>> Best regards,
>>> Kien
>>> 


Mime
View raw message