flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Window stream using timestamp key for time
Date Thu, 28 Jan 2016 22:34:58 GMT
Hi Emmanuel,

the feature you are looking for is called event time processing in Flink.
These blog posts should help you to become familiar with the concepts:

1) Event-Time concepts:
2) Windows in Flink:
3) Event-Time example use-case:
4) Code for example: https://github.com/dataArtisans/flink-streaming-demo

Best, Fabian

2016-01-28 23:08 GMT+01:00 Emmanuel <eleroy@msn.com>:

> Hello,
> I have used Flink to stream data and do analytics on the stream, using
> time windows...
> Now, this is assuming the data is effectively coming in real time. However
> I have a use case where the data is 'batched' upstream, and comes in
> bursts, but has a timestamp.
> It obviously messes up the windowed stream assumption.
> (note it is a problem with queuing in Kafka for example when there is any
> kind of downtime downstream of Kafka: if data accumulates and then is
> consumed, it is consumed at higher 'speed' than real clock time and
> statistics do not match reality.)
> So my question is:
> Is it possible to use a window stream based on a timestamp key for time,
> as opposed to clock time?
> How would one do this with the current API?
> Thanks
> Emmanuel

View raw message