flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: State in Custom Tumble Window Class
Date Tue, 06 Jun 2017 14:21:17 GMT
Hi,
Yes I think you can manually store the latest watermark by using the OperatorStateStore that
you get if your user function implements the CheckpointedFunction interface.

Best,
Aljoscha
> On 30. May 2017, at 13:43, rhashmi <rizhashmi@hotmail.com> wrote:
> 
> Thanks Aljoscha Krettek, 
> 
> So the results will not be deterministic for late events. For idempotent
> update, i would need to find an additional key base of current event time if
> they are late and attached to the aggregator which probably possible by
> doing some function(maxEventTime, actualEventTime). For that i need
> maxEventTime to be stored as part of state & recover in case of runtime
> failure.
> 
> Here is my corner case like. 
> -- If assume whole flink runtime crashed(auto commit on) & after recovery
> the first event arrived is from past(actually late). Without keeping max
> currentTime state, may potentially override previous aggregate. 
> 
> I was wondering if i can record my last max EventTime as part of checkPoint, 
> or run query against sink source to find last processed eventtime.
> 
> Any recommendation?
> 
> 
> 
> 
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/State-in-Custom-Tumble-Window-Class-tp13177p13387.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.


Mime
View raw message