flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Window on stream with timestamps ascending by key
Date Sat, 19 Mar 2016 11:48:01 GMT
Hi,
what you essentially would require is watermarks that are tracked by key. Right now this is
not possible in Flink. The watermarks, which are used for keeping track of the timestamps,
are global across all keys.

Maybe you could implement something that fits your requirements in a custom operator, i.e.
by using DataStream.transform() and writing a StreamOperator (more specifically a OneInputStreamOperator).

Let us know if you need more information.

Cheers,
Aljoscha
> On 17 Mar 2016, at 11:21, Charles-Antoine Mathieu <charles-antoine.mathieu@corp.ovh.com>
wrote:
> 
> I wonder how to work with a stream with event timestamps ascending by key.
> 
> I can have a huge time skew between different keys, for example if I (re)connect an event
producer,
> it will send all buffered results possibly from the last days.
> 
> Is it possible to trigger the window computation per key ?
> 
> Example with a window of 5 seconds and the window function being the count of the timestamps
:
> 
> KEY1 1000
> KEY1 1001
> KEY1 1002
> KEY2    1
> KEY2    2
> KEY2    3
> KEY2    4
> KEY2    5
> KEY2 window => 15
> KEY1 1003
> KEY2    6
> KEY2    7
> KEY2    8
> KEY2    9
> KEY2   10
> KEY2 window => 40
> KEY1 1004
> KEY2   11
> KEY2   12
> KEY2   13
> KEY2   14
> KEY2   15
> KEY2 window => 65
> KEY1 1005
> KEY1 window => 5015
> ...
> 
> 


Mime
View raw message