flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: Cannot see all events in window apply() for big input
Date Tue, 08 Nov 2016 18:00:18 GMT
Flink does not support per key watermarks or type sensitive watermarks. The
underlying assumption is that you have a global watermark which defines the
progress wrt to event time in your topology.

The easiest way would be to have an input which has a monotonically
increasing timestamp. Alternatively you can define the maximum lag between
the watermark and the timestamp and then generate watermarks with w =
timestamp - maxLag. That way you allow elements to be out of order for a
certain amount of event time.

Cheers,
Till

On Tue, Nov 8, 2016 at 5:02 PM, Sendoh <unicorn.banachi@gmail.com> wrote:

> Thank you for confirming.
>
> What would you think an efficient way not having global watermark? The
> following logic fails to build Watermark per KeyStream:
> jsonStreams.keyBy(new JsonKeySelector()).assignTimestampsAndWatermarks(new
> JsonWatermark()).keyBy(JsonKeySelector()).window(....
>
> So, using split(), or implementing an event type recognized
> AssignerWithPeriodicWatermarks along with custom EventTimeTrigger would be
> the solution?
>
> Best,
>
> Sendoh
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Cannot-see-all-
> events-in-window-apply-for-big-input-tp9945p9988.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Mime
View raw message