flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Window PURGE Behaviour 1.0.2 vs 1.1.3
Date Thu, 10 Nov 2016 13:48:04 GMT
Hi,
there were some discussions on the ML and it seems that the consensus is to
aim for a release this year.

Let me think a bit more and get back to you on the other issues.

Cheers,
Aljoscha

On Thu, 10 Nov 2016 at 11:47 Konstantin Knauf <konstantin.knauf@tngtech.com>
wrote:

> Hi Aljoscha,
>
> unfortunately, I think, FLINK-4994 would not solve our issue. What does
> "on the very end" mean in case of a GlobalWindow?
>
> FLINK-4369 would fix my workaround though. Is there already a timeline
> for Flink 1.2?
>
> Cheers,
>
>
> Konst
>
> On 10.11.2016 10:19, Aljoscha Krettek wrote:
> > Hi Konstantin,
> > evicting elements not being evicted is a bug that should be fixed for
> > Flink 1.2: https://issues.apache.org/jira/browse/FLINK-4369.
> >
> > The check about non-existing window state when triggering was introduced
> > because otherwise a Trigger could return FIRE and then there would be
> > nothing to fire. I guess if we did indeed fire the trigger even with
> > non-existing state then some people might wonder why no emission is
> > triggered when their trigger returns FIRE. I see your point though, that
> > the omitted firing is problematic for some cases.
> >
> > I think having clear() as proposed
> > in https://issues.apache.org/jira/browse/FLINK-4994 would solve your
> > case. You were using your own cleanup timer as a workaround because
> > clear() is currently also called on PURGE. With clear() only being
> > called at the very end this should work, correct?
> >
> > Cheers,
> > Aljoscha
> >
> > On Wed, 9 Nov 2016 at 19:53 Konstantin Knauf
> > <konstantin.knauf@tngtech.com <mailto:konstantin.knauf@tngtech.com>>
> wrote:
> >
> >     Hi Aljoscha,
> >
> >     as it turns out the "workaround" I was thinking was functionally
> >     working, but had a so to say memory leak. I was under the impression
> >     that evicted elements will be removed from the window state...
> >
> >     Anyway, I think that this (triggers not being evaluated when the
> window
> >     state is null) turns out to be blocker for us.
> >
> >     Why is this check done? Since a user can do basically whatever she
> likes
> >     in onProcessingTimeTimer() the comment
> >
> >     // if we have no state, there is nothing to do
> >
> >     is, well, just not true in some cases (e.g. state updates in our
> case).
> >
> >     Cheers,
> >
> >     Konstantin
> >
> >     On 09.11.2016 14:17, Aljoscha Krettek wrote:
> >     > Could you go into some detail of why you need to keep the trigger
> >     state?
> >     >
> >     > Just the basics because you probably cannot (should not) talk
> >     about your
> >     > internal stuff.
> >     >
> >     > On Wed, 9 Nov 2016 at 13:16 Konstantin Knauf
> >     > <konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>
> >     <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>>> wrote:
> >     >
> >     >     Sounds good Aljoscha.
> >     >
> >     >     sent from my phone. Plz excuse brevity and tpyos.
> >     >     ---
> >     >     Konstantin Knauf *konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>
> >     >     <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>> * +49-174-3413182
> <0174%203413182>
> >     <tel:0174%203413182>
> >     >     <tel:0174%203413182>
> >     >
> >     >     TNG Technology Consulting GmbH, Betastr. 13a, 85774
> Unterföhring
> >     >     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert
> >     Dahlke
> >     >
> >     >     ---- Aljoscha Krettek schrieb ----
> >     >
> >     >
> >     >     Hi,
> >     >     exactly for this case I want to make a change to when
> >     >     Trigger.clear() is
> >     >     called: https://issues.apache.org/jira/browse/FLINK-4994
> >     >
> >     >     Right now, clear is called when the window is being garbage
> >     >     collected because we passed the allowed lateness (after this,
> >     >     nothing will ever be added to a window again) and also when the
> >     >     Trigger returns PURGE or FIRE_AND_PURGE.
> >     >
> >     >     I want to change it to only be called in the former case. We
> could
> >     >     possibly add an onPurge() callback to allow cleaning state on
> >     purge
> >     >     or require people to put the code that they want to run on
> >     PURGE in
> >     >     the Trigger method that returns the PURGE.
> >     >
> >     >     What do you think?
> >     >
> >     >     Cheers,
> >     >     Aljoscha
> >     >
> >     >     On Tue, 8 Nov 2016 at 18:46 Konstantin Knauf
> >     >     <konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>
> >     <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>>>
> >     >     wrote:
> >     >
> >     >         Hi Aljoscha,
> >     >
> >     >         interesting, this explains it. Well, in our case the PURGE
> >     in the
> >     >         onProcessingTimeTimer is only used to clear
> >     KeyValueStates*, and
> >     >         at this
> >     >         point there are usually no records in the window state.
> >     >
> >     >         Any Ideas?
> >     >
> >     >         I do have a workaround with an evictor, but it seemed to be
> >     >         unnecessarily complicated.
> >     >
> >     >         *We can not use clear()-callback for that, since this
> >     state should
> >     >         survive the FIRE_AND_PURGEs in the onElement()-calls.
> >     >
> >     >         Cheers,
> >     >
> >     >         Konstantin
> >     >
> >     >
> >     >         On 08.11.2016 18:31, Aljoscha Krettek wrote:
> >     >         > Hi,
> >     >         > the timers are not actually deleted but the
> WindowOperator
> >     >         will check
> >     >         > whether there is any window state associated with the
> window
> >     >         for which
> >     >         > the timer fires. If there is no window state the timer
> will
> >     >         silently be
> >     >         > ignored.
> >     >         >
> >     >         > Is this a problem for you or did you just want to
> >     clarify? If
> >     >         yes, then
> >     >         > we should work on finding a solution.
> >     >         >
> >     >         > Cheers,
> >     >         > Aljoscha
> >     >         >
> >     >         > On Tue, 8 Nov 2016 at 18:18 Konstantin Knauf
> >     >         > <konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>
> >     >         <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>>
> >     >         <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>
> >     >         <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>>>> wrote:
> >     >         >
> >     >         >     Hi everyone,
> >     >         >
> >     >         >     I just migrated a streaming Job from 1.0.2 to 1.1.3
> and
> >     >         stumbled across
> >     >         >     a problem concerning one of our custom triggers.
> >     >         >
> >     >         >     The trigger basically FIRE_AND_PURGEs multiple times
> in
> >     >         onElement() and
> >     >         >     the window is PURGEd onProcessingTimeTimer(), but it
> >     seems
> >     >         that the all
> >     >         >     registered processing time timers are deleted
> everytime
> >     >         the window is
> >     >         >     PURGEd.
> >     >         >
> >     >         >     clear() is the default implementation, i.e. no-op.
> >     >         >
> >     >         >     Just wanted to, if this is the expected behavior
> >     >         (processing time timers
> >     >         >     being deleted on PURGE or FIRE_AND_PURGE) from Flink
> >     1.1 on?
> >     >         >
> >     >         >     Cheers,
> >     >         >
> >     >         >     Konstantin
> >     >         >
> >     >         >     --
> >     >         >     Konstantin Knauf * konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>
> >     >         <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>>
> >     >         >     <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>
> >     >         <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>>> * +49-174-3413182
> <0174%203413182>
> >     <tel:0174%203413182>
> >     >         <tel:0174%203413182>
> >     >         >     <tel:0174%203413182>
> >     >         >     TNG Technology Consulting GmbH, Betastr. 13a, 85774
> >     >         Unterföhring
> >     >         >     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr.
> >     >         Robert Dahlke
> >     >         >     Sitz: Unterföhring * Amtsgericht München * HRB 135082
> >     >         >
> >     >         >
> >     >
> >     >         --
> >     >         Konstantin Knauf * konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>
> >     >         <mailto:konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com>> * +49-174-3413182
> <0174%203413182>
> >     <tel:0174%203413182>
> >     >         <tel:0174%203413182>
> >     >         TNG Technology Consulting GmbH, Betastr. 13a, 85774
> >     Unterföhring
> >     >         Geschäftsführer: Henrik Klagges, Christoph Stock, Dr.
> >     Robert Dahlke
> >     >         Sitz: Unterföhring * Amtsgericht München * HRB 135082
> >     >
> >
> >     --
> >     Konstantin Knauf * konstantin.knauf@tngtech.com
> >     <mailto:konstantin.knauf@tngtech.com> * +49-174-3413182
> <0174%203413182>
> >     <tel:0174%203413182>
> >     TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> >     Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> >     Sitz: Unterföhring * Amtsgericht München * HRB 135082
> >
>
> --
> Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
> <0174%203413182>
> TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
> Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
> Sitz: Unterföhring * Amtsgericht München * HRB 135082
>
>

Mime
View raw message