flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "G.S.Vijay Raajaa" <gsvijayraa...@gmail.com>
Subject Re: Purging Late stream data
Date Wed, 26 Jul 2017 10:55:04 GMT
Sure, Let me try that out. On the same note, does
BoundedOutOfOrdernessTimestampExtractor
Serve the purpose too?


Regards,
Vijay Raajaa GS

On Wed, Jul 26, 2017 at 9:22 AM, Kien Truong <duckientruong@gmail.com>
wrote:

> Hi,
>
> One method you can use is using a ProcessFunction.
>
> In the process function, you get the timer service through the function
> context,
>
> which can then be used to schedule a task to clean up late data.
>
> Check out the docs for ProcessFunction
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
> dev/stream/process_function.html
>
> Regards,
>
> Kien
>
>
>
> On 7/26/2017 9:37 AM, G.S.Vijay Raajaa wrote:
>
>> Hi,
>>
>> I am having 3 streams which is being merged from a union of kafka topics
>> on a given timestamp. The problem I am facing is that, if there is a delay
>>  in one of the stream and when the data in that particular stream arrives
>> at a later point in time, the merge happens in a delayed fashion. The way I
>> want to solve is that, I want to drop such data streams which comes after a
>> delay ( say 5sec ). Kindly direct me on how to go about it?
>>
>> Will watermarking (to process in even time) + the allowed lateness help
>> solve the same?
>>
>> Regards,
>> Vijay Raajaa G S
>>
>
>

Mime
View raw message