flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sujit Sakre <sujit.sa...@northgateps.com>
Subject Inconsistent Results in Event Time based Sliding Window processing
Date Tue, 14 Feb 2017 11:27:31 GMT
Hi,

I have been using Flink 1.1.1 with Kafka 0.9 to process real time streams.
We have written our own Window Function and are processing data with
Sliding Windows. We are using Event Time and use a custom watermark
generator.

We select a particular window out of multiple sliding windows and process
all items in the window in an iterative loop to increment counts of the
items selected. After this we call the sink method to log the result in a
database.

This is working fine most of the times, i.e. it produces the expected
result most of the times. However there are situations when certain windows
are not processed (under same test conditions), this means the results are
less than expected i.e. the counts are less than expected. Sometimes, items
instead of certain windows not getting processed, certain items do not get
processed in the window. This is unpredictable.

I wish to know what could be the cause of this inconsistent behaviour. How
do we resolve it. We have integrated 1.2 and Kafka 0.10 now, the problem
persists.

Please could you suggest about what the problem could be and how to resolve
this.

Many thanks.


*Sujit Sakre*

-- 
This email is sent on behalf of Northgate Public Services (UK) Limited and 
its associated companies including Rave Technologies (India) Pvt Limited 
(together "Northgate Public Services") and is strictly confidential and 
intended solely for the addressee(s). 
If you are not the intended recipient of this email you must: (i) not 
disclose, copy or distribute its contents to any other person nor use its 
contents in any way or you may be acting unlawfully;  (ii) contact 
Northgate Public Services immediately on +44(0)1908 264500 quoting the name 
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that 
no viruses are contained in this email, but does not accept any 
responsibility once this email has been transmitted.  You should scan 
attachments (if any) for viruses.

Northgate Public Services (UK) Limited, registered in England and Wales 
under number 00968498 with a registered address of Peoplebuilding 2, 
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2 
4NN.  Rave Technologies (India) Pvt Limited, registered in India under 
number 117068 with a registered address of 2nd Floor, Ballard House, Adi 
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.

Mime
View raw message