apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav Gupta <gau...@datatorrent.com>
Subject Re: Managing state within an application window
Date Mon, 09 Nov 2015 08:27:32 GMT

Platform supports sliding window feature natively. You can refer to SLIDE_BY_WINDOW_COUNT
attribute (https://github.com/apache/incubator-apex-core/blob/master/api/src/main/java/com/datatorrent/api/Context.java
and you can also look at SliderTest https://github.com/apache/incubator-apex-core/blob/master/engine/src/test/java/com/datatorrent/stram/engine/SliderTest.java

- Gaurav

> On Nov 9, 2015, at 12:00 AM, Bhupesh Chawda <bhupesh@datatorrent.com> wrote:
> Yes, it will be reset in the endWindow call. So nothing should be needed to
> be done explicitly.
> Also, after the crash, the operator is restored to the checkpoint which had
> happened after beginWindow but before endWindow. In this case the
> beginWindow() call will be skipped when the operator restarts after
> recovery.
> It is easy to wrongly assume (which I did) that checkpoints always happen
> "after endWindow" (in which case, the variable can be made transient).
> I also had a concern regarding *sliding application windows*. It is not
> clear as to how the beginWindow and endWindow calls would be structured. In
> addition to the application window size, we would also need some *delta*
> which is the amount by which the window would "slide" by. How is this
> specified? Is this per tuple? Also how would checkpointing and recovery act
> in such a case?
> Thanks.
> Bhupesh
> On Mon, Nov 9, 2015 at 12:39 PM, Milind Barve <milindb@gmail.com> wrote:
>> You might want to initialize it in beginwidow() as well. Should keep it non
>> transient so for recovery.
>> On Nov 9, 2015 12:34 PM, "Amol Kekre" <amol@datatorrent.com> wrote:
>>> Bhupesh.
>>> I would recommend reseting the variable in endWindow call and leaving it
>>> non-transient. Usually an empty variable should not add too much to
>>> serialization.
>>> Thks,
>>> Amol
>>> On Sun, Nov 8, 2015 at 10:57 PM, Bhupesh Chawda <bhupesh@datatorrent.com
>>> wrote:
>>>> Hi All,
>>>> I have a question regarding maintaining state of a variable within a
>>> single
>>>> application window.
>>>> If I have a variable which is just needed for a single application
>> window
>>>> (just between beginWindow and endWindow calls), do I need to serialize
>>> and
>>>> checkpoint it? or can I make it transient?
>>>> I understand that in case of checkpoint windows being aligned with the
>>>> application windows, we can do so. Since in this case, the variable
>> will
>>>> start afresh immediately after the checkpoint. However in case of
>>>> checkpoints happening in between application windows, this might not be
>>> the
>>>> case.
>>>> In the second case, is there any mechanism using which I can
>>> conditionally
>>>> make it transient and non-transient based on when the checkpoints
>> happen
>>>> with respect to the application windows?
>>>> Thanks.
>>>> Bhupesh

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message