flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nico Kruber <n...@data-artisans.com>
Subject Re: Inconsistent Results in Event Time based Sliding Window processing
Date Tue, 14 Feb 2017 13:25:46 GMT
Hi Sujit,
this does indeed sound strange and we are not aware of any data loss issues.
Are there any exceptions or other errors in the job/taskmanager logs?
Do you have a minimal working example? Is it that whole windows are not 
processed or just single items inside a window?


Nico

On Tuesday, 14 February 2017 16:57:31 CET Sujit Sakre wrote:
> Hi,
> 
> I have been using Flink 1.1.1 with Kafka 0.9 to process real time streams.
> We have written our own Window Function and are processing data with
> Sliding Windows. We are using Event Time and use a custom watermark
> generator.
> 
> We select a particular window out of multiple sliding windows and process
> all items in the window in an iterative loop to increment counts of the
> items selected. After this we call the sink method to log the result in a
> database.
> 
> This is working fine most of the times, i.e. it produces the expected
> result most of the times. However there are situations when certain windows
> are not processed (under same test conditions), this means the results are
> less than expected i.e. the counts are less than expected. Sometimes, items
> instead of certain windows not getting processed, certain items do not get
> processed in the window. This is unpredictable.
> 
> I wish to know what could be the cause of this inconsistent behaviour. How
> do we resolve it. We have integrated 1.2 and Kafka 0.10 now, the problem
> persists.
> 
> Please could you suggest about what the problem could be and how to resolve
> this.
> 
> Many thanks.
> 
> 
> *Sujit Sakre*


Mime
View raw message