flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kien Truong <duckientru...@gmail.com>
Subject Re: Purging Late stream data
Date Wed, 26 Jul 2017 03:52:24 GMT
Hi,

One method you can use is using a ProcessFunction.

In the process function, you get the timer service through the function 
context,

which can then be used to schedule a task to clean up late data.

Check out the docs for ProcessFunction

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/process_function.html

Regards,

Kien


On 7/26/2017 9:37 AM, G.S.Vijay Raajaa wrote:
> Hi,
>
> I am having 3 streams which is being merged from a union of kafka 
> topics on a given timestamp. The problem I am facing is that, if there 
> is a delay  in one of the stream and when the data in that particular 
> stream arrives at a later point in time, the merge happens in a 
> delayed fashion. The way I want to solve is that, I want to drop such 
> data streams which comes after a delay ( say 5sec ). Kindly direct me 
> on how to go about it?
>
> Will watermarking (to process in even time) + the allowed lateness 
> help solve the same?
>
> Regards,
> Vijay Raajaa G S


Mime
View raw message