flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radu Tudoran <radu.tudo...@huawei.com>
Subject RE: question about rowtime processfunction - are watermarks needed?
Date Thu, 27 Apr 2017 14:21:20 GMT
Thanks for the explanation.

-----Original Message-----
From: Fabian Hueske [mailto:fhueske@gmail.com] 
Sent: Thursday, April 27, 2017 3:17 PM
To: dev@flink.apache.org
Subject: Re: question about rowtime processfunction - are watermarks needed?

Hi Radu,

event-time processing requires watermarks. Operators use watermarks to compute the current
event-time.
The ProcessFunctions for over range windows use the TimerServices to group elements by time.
In case of event-time, the timers are triggered by the event-time of the operator which is
derived from the received watermarks.
In case of processing-time, the timers are triggered based on the wallclock time of the operator.

So by using event-tim timers, we implicitly rely on the watermarks because the timers are
triggered based on the received watermarks.

Best, Fabian


2017-04-27 10:51 GMT+02:00 Radu Tudoran <radu.tudoran@huawei.com>:

> Hi,
>
> I am looking at the implementation of  RowTimeBoundedRangeOver (in the 
> context of Stream SQL). I see that the logic is that the progress 
> happens based on the timestamps of the rowevent - i.e., when an even 
> arrives we register to be processed based on it's timestamp (ctx.timerService.
> registerEventTimeTimer(triggeringTs))
>
> In the onTimer we remove (retract) data that has expired. However, we 
> do not consider watermarks nor some allowed latency for the events or 
> anything like this, which makes me ask:
> Don't we need to work with watermarks when we deal with even time? And 
> keep the events within the allowed delayed/next watermark?  Am I 
> missing something? Or maybe we do not consider at this point 
> allowedLateness  for this version?
>
> Thanks
>
> Best regards,
>
>
Mime
View raw message