flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Knauf <konstantin.kn...@tngtech.com>
Subject Re: [DISCUSS] Removing delete*Timer from the WindowOperator.Context
Date Wed, 12 Oct 2016 16:05:47 GMT
Hi all,

thank you for looping me in. Because of the memory leak we first
experienced we have built a work-around, which did not need to delete
timers and are still using it. So for us, I think, this would currently
not be a problem. Nevertheless, I think, it is a strong limitation if
custom triggers can not delete timers. I am not familiar with the new
Trigger DSL though.

Cheers,

Konstantin

On 12.10.2016 15:38, Kostas Kloudas wrote:
> Hi all,
> 
> This thread has been dormant for some time now.
> 
> Given that this change may affect user code, I am sending this as a 
> reminder that the discussion is still open and to re-invite anyone who
> may be affected to participate.
> 
> I would suggest to leave it open till the end of next week and then, 
> if nobody objects, we can proceed to the change.
> 
> What do you think?
> 
> Kostas
> 
>> On Sep 28, 2016, at 3:21 PM, Maximilian Michels <mxm@apache.org> wrote:
>>
>> What are the use cases where you actually need to delete a timer? How
>> about we only let users delete timers which they created themselves?
>>
>> I guessing most of these use cases will be obsolete with the new
>> Trigger DSL because the trigger logic can be expressed more easily. So
>> +1 for removing the delete methods from the context.
>>
>> On Tue, Sep 27, 2016 at 3:43 PM, Kostas Kloudas
>> <k.kloudas@data-artisans.com> wrote:
>>> Hi all,
>>>
>>> As the title of this email suggests, I am proposing to remove the  methods
>>> deleteProcessingTimeTimer(long time) and deleteEventTimeTimer(long time)
>>> from the WindowOperator.Context. With this change, registered timers that
>>> have nothing to do (e.g. because their state has already been cleaned up)
>>> will be simply ignored by the windowOperator, when their time comes.
>>>
>>> The reason for the change is that by allowing custom user code, e.g. a custom
Trigger,
>>> to delete timers we may have unpredictable behavior.
>>>
>>> As an example, one can imagine the case where we have allowed_lateness = 0 and
the cleanup
>>> timer for a window collides with the end_of_window one. In this case, by deleting
the end_of_window
>>> timer from the trigger (possibly a custom one), we end up also deleting the cleanup
one,
>>> which in turn can lead to the window state never being garbage collected.
>>>
>>> To see what can be the consequences apart from memory leaks, this can easily
lead
>>> to wrong session windows, as a session that should have been garbage collected,
will
>>> still be around and ready to accept new data.
>>>
>>> With this change, timers that should correctly be deleted will now remain in
the queue of
>>> pending timers, but they will do nothing, while cleanup timers will cleanup the
state of their
>>> corresponding window.
>>>
>>> Other possible solutions like keeping a separate list for cleanup timers would
complicate
>>> the codebase and also introduce memory overheads which can be avoided using the
>>> solution above (i.e. just ignoring timers the have nothing to do anymore).
>>>
>>> What do you think?
>>>
>>> Kostas
>>>
> 
> 

-- 
Konstantin Knauf * konstantin.knauf@tngtech.com * +49-174-3413182
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterföhring
Geschäftsführer: Henrik Klagges, Christoph Stock, Dr. Robert Dahlke
Sitz: Unterföhring * Amtsgericht München * HRB 135082


Mime
View raw message