flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominik Bruhn <domi...@dbruhn.de>
Subject BoundedOutOfOrdernessTimestampExtractor and timestamps in the future
Date Tue, 01 Nov 2016 14:05:25 GMT
Hey,
I'm using a BoundedOutOfOrdernessTimestampExtractor for assigning my 
timestamps and discarding to old events (which happens sometimes).

Now my problem is that some events, by accident have timestamps in the 
future. If the timestamps are more in the future than my 
`maxOutOfOrderness`, I'm discarding valid events. So I need a way of 
saying that the
BoundedOutOfOrdernessTimestampExtractor should exclude timestamps from 
the future for the watermark calculation. I still want to keep the 
events if they are in the future and assign them to the right 
watermarks.

How can I achieve this? I thought about checking whether the potential 
timestamp is in the future before considering it for a watermark. I 
cloned the BoundedOutOfOrdernessTimestampExtractor and added the idea
https://gist.github.com/theomega/090054032e0b3c3b9fb06767f0fec7e7

Does this make sense? Or is there a better approach?

In general, how does Flink handle readings from the future?

Thanks,
Dominik

-- 
Dominik

Mime
View raw message