flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Petr Novotnik <petr.novot...@firma.seznam.cz>
Subject WindowOperator - element's timestamp
Date Fri, 11 Nov 2016 15:16:58 GMT
Hello,

I'm struggling to understand the following behaviour of the 
`WindowOperator` and would appreciate some insight from experts:

In particular I'm thinking about the following hypothetical data flow:

input.keyBy(..)
      .window(TumblingEventTimeWindows.of(..))
      .apply(..)
      ...
      .keyBy(..)
      .window(CustomWindowAssignerUsingATriggerBasedOnTheElementsStamp)
      .apply(..)

When the first window operator fires a window based on the timer, the 
emitted elements are assigned a timestamp which equals 
`window.maxTimestamp()`. This stamp is then available in the second 
window operator's trigger through the `onElement` method. So far so good.

However, when using `ContinuousEventTimeTrigger` (simply put when firing 
the window multiple times at different times in its lifecycle) in the 
first window operator, _all_ of the elements of this window - no matter 
whether fired as a partial or the final window result - will arrive with 
the same stamp in the (downstream) operators.

This make it practically impossible to use again 
`ContinuousEventTimeTrigger` (or similar) in the second window operator 
to achieve "early firing" again.

This is surprising. I would expect the elements to be assigned the stamp 
of the timer which fired them (which will be window#maxTimestamp() for 
`TumblingEventTimeWindows`). Is there any particular reason for the 
unconditional assignment to `window.maxTimestamp()`?

Many thanks in advance,
P.

Mime
View raw message