flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: TimeWindow Trigger which only fires when the values have changed
Date Tue, 04 Oct 2016 13:08:47 GMT
Hi Dominik,

you could extend the EventTimeTriggerWithEarlyAndLateFiring trigger to
store for each key whether you’ve seen a new element since the last firing
or not. When firing you can set the state back to alreadyFired. For that
you can use the TriggerContext.getPartitionedState.

The community is currently on reworking Flink’s triggers to provide a
trigger DSL [1]. With this DSL you’ll be able to specify a trigger as
you’ve found with EventTimeTriggerWithEarlyAndLateFiring.

Cheers,
Till
​

On Tue, Oct 4, 2016 at 2:54 PM, Dominik Bruhn <dominik@dbruhn.de> wrote:

> Hi,
> I'm heavily relying on TimeWindows for my real time processing. Roughly my
> job consumes from an AMQP queue, computes some time buckets and saves the
> time-buckets to Cassandra.
>
> I found the EventTimeTriggerWithEarlyAndLateFiring [1] class which
> already helped me a lot: Even with long time-windows, I can get
> intermediate values already saved to Cassandra by using the earlyFiring
> (and setting "accumulating" to true.
>
> My question is: Would it be possible to only write fire the trigger if the
> value of the TimeBucket has changed? What I actually want is only writing
> to Cassandra if there is actually something different in the time bucket.
>
> And, as a side question: Why is something like the
> EventTimeTriggerWithEarlyAndLateFiring not in the default Flink
> distribution? It seems very handy.
>
> [1]: https://github.com/dataArtisans/beam_comp/blob/master/src/
> main/java/com/dataartisans/beam_comparison/customTriggers
> /EventTimeTriggerWithEarlyAndLateFiring.java
>
>
> Thanks,
> Dominik
>

Mime
View raw message