flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kien Truong <duckientru...@gmail.com>
Subject Re: Purging Late stream data
Date Wed, 26 Jul 2017 03:52:24 GMT

One method you can use is using a ProcessFunction.

In the process function, you get the timer service through the function 

which can then be used to schedule a task to clean up late data.

Check out the docs for ProcessFunction




On 7/26/2017 9:37 AM, G.S.Vijay Raajaa wrote:
> Hi,
> I am having 3 streams which is being merged from a union of kafka 
> topics on a given timestamp. The problem I am facing is that, if there 
> is a delay  in one of the stream and when the data in that particular 
> stream arrives at a later point in time, the merge happens in a 
> delayed fashion. The way I want to solve is that, I want to drop such 
> data streams which comes after a delay ( say 5sec ). Kindly direct me 
> on how to go about it?
> Will watermarking (to process in even time) + the allowed lateness 
> help solve the same?
> Regards,
> Vijay Raajaa G S

View raw message