Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Subject: Re: Purging Late stream data
To: "G.S.Vijay Raajaa" <gsvijayraajaa@gmail.com>, user <user@flink.apache.org>
References: <CAB9NB9ab_CM9iU17vBUOKAc+u_qHoqZssjWUBCm5RZeMYN_ejA@mail.gmail.com>
From: Kien Truong <duckientruong@gmail.com>
Message-ID: <d2c3e2a9-3677-4442-5875-ef0c7370db73@gmail.com>
Date: Wed, 26 Jul 2017 10:52:24 +0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:55.0) Gecko/20100101
 Thunderbird/55.0
MIME-Version: 1.0
In-Reply-To: <CAB9NB9ab_CM9iU17vBUOKAc+u_qHoqZssjWUBCm5RZeMYN_ejA@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
archived-at: Wed, 26 Jul 2017 03:52:38 -0000

Hi,

One method you can use is using a ProcessFunction.

In the process function, you get the timer service through the function 
context,

which can then be used to schedule a task to clean up late data.

Check out the docs for ProcessFunction

https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/process_function.html

Regards,

Kien


On 7/26/2017 9:37 AM, G.S.Vijay Raajaa wrote:
> Hi,
>
> I am having 3 streams which is being merged from a union of kafka 
> topics on a given timestamp. The problem I am facing is that, if there 
> is a delay  in one of the stream and when the data in that particular 
> stream arrives at a later point in time, the merge happens in a 
> delayed fashion. The way I want to solve is that, I want to drop such 
> data streams which comes after a delay ( say 5sec ). Kindly direct me 
> on how to go about it?
>
> Will watermarking (to process in even time) + the allowed lateness 
> help solve the same?
>
> Regards,
> Vijay Raajaa G S