flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Window PURGE Behaviour 1.0.2 vs 1.1.3
Date Wed, 09 Nov 2016 10:49:05 GMT
Hi,
exactly for this case I want to make a change to when Trigger.clear() is
called: https://issues.apache.org/jira/browse/FLINK-4994

Right now, clear is called when the window is being garbage collected
because we passed the allowed lateness (after this, nothing will ever be
added to a window again) and also when the Trigger returns PURGE or
FIRE_AND_PURGE.

I want to change it to only be called in the former case. We could possibly
add an onPurge() callback to allow cleaning state on purge or require
people to put the code that they want to run on PURGE in the Trigger method
that returns the PURGE.

What do you think?

Cheers,
Aljoscha

On Tue, 8 Nov 2016 at 18:46 Konstantin Knauf <konstantin.knauf@tngtech.com>
wrote:

> Hi Aljoscha,
>
> interesting, this explains it. Well, in our case the PURGE in the
> onProcessingTimeTimer is only used to clear KeyValueStates*, and at this
> point there are usually no records in the window state.
>
> Any Ideas?
>
> I do have a workaround with an evictor, but it seemed to be
> unnecessarily complicated.
>
> *We can not use clear()-callback for that, since this state should
> survive the FIRE_AND_PURGEs in the onElement()-calls.
>
> Cheers,
>
> Konstantin
>
>
> On 08.11.2016 18:31, Aljoscha Krettek wrote:
> > Hi,
> > the timers are not actually deleted but the WindowOperator will check
> > whether there is any window state associated with the window for which
> > the timer fires. If there is no window state the timer will silently be
> > ignored.
> >
> > Is this a problem for you or did you just want to clarify? If yes, then
> > we should work on finding a solution.
> >
> > Cheers,
> > Aljoscha
> >
> > On Tue, 8 Nov 2016 at 18:18 Konstantin Knauf
> > <konstantin.knauf@tngtech.com <mailto:konstantin.knauf@tngtech.com>>
> wrote:
> >
> >     Hi everyone,
> >
> >     I just migrated a streaming Job from 1.0.2 to 1.1.3 and stumbled
> across
> >     a problem concerning one of our custom triggers.
> >
> >     The trigger basically FIRE_AND_PURGEs multiple times in onElement()
> and
> >     the window is PURGEd onProcessingTimeTimer(), but it seems that the
> all
> >     registered processing time timers are deleted everytime the window is
> >     PURGEd.
> >
> >     clear() is the default implementation, i.e. no-op.
> >
> >     Just wanted to, if this is the expected behavior (processing time
> timers
> >     being deleted on PURGE or FIRE_AND_PURGE) from Flink 1.1 on?
> >
> >     Cheers,
> >
> >     Konstantin
> >
> >     --
> >     Konstantin Knauf * konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com> * +49-174-3413182
> <0174%203413182>
> >     <tel:0174%203413182>
> >     TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> >     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> >     Sitz: Unterföhring * Amtsgericht München * HRB 135082
> >
> >
>
> --
> Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
> <0174%203413182>
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

Mime
View raw message