Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.incubator.apache.org
From: Gaurav Gupta <gaurav@datatorrent.com>
Content-Type: multipart/alternative;
 boundary="Apple-Mail=_A7870865-0D08-4E30-99ED-3416EF5ED5A1"
Message-Id: <6D68F8A7-2150-42D3-A9DA-72453AD8B57F@datatorrent.com>
Mime-Version: 1.0 (Mac OS X Mail 9.0 \(3094\))
Subject: Re: Managing state within an application window
Date: Mon, 9 Nov 2015 00:27:32 -0800
References: 
 <CAHvM9d3SDmdD57PygMtvJ4EvuX6aUTtHA0MOR_tZ5T1waCdv9g@mail.gmail.com>
 <CAMaZnT4NqgfUxD1au0V9_GHQ_svob=F70g4UFo9N7vf5BL85fg@mail.gmail.com>
 <CAB2ZaZPNduSiO3GaYj1qmuASnU0Sx0nM+e_zOdu1LU-nxT6EvA@mail.gmail.com>
 <CAHvM9d2DnhrGUogXHw0J8XWsxyYu89pF1uocTheLbURt97S0fw@mail.gmail.com>
To: dev@apex.incubator.apache.org
In-Reply-To: 
 <CAHvM9d2DnhrGUogXHw0J8XWsxyYu89pF1uocTheLbURt97S0fw@mail.gmail.com>

--Apple-Mail=_A7870865-0D08-4E30-99ED-3416EF5ED5A1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Bhupesh,

Platform supports sliding window feature natively. You can refer to =
SLIDE_BY_WINDOW_COUNT attribute =
(https://github.com/apache/incubator-apex-core/blob/master/api/src/main/ja=
va/com/datatorrent/api/Context.java =
<https://github.com/apache/incubator-apex-core/blob/master/api/src/main/ja=
va/com/datatorrent/api/Context.java>) and you can also look at =
SliderTest =
https://github.com/apache/incubator-apex-core/blob/master/engine/src/test/=
java/com/datatorrent/stram/engine/SliderTest.java

Thanks
- Gaurav

> On Nov 9, 2015, at 12:00 AM, Bhupesh Chawda <bhupesh@datatorrent.com> =
wrote:
>=20
> Yes, it will be reset in the endWindow call. So nothing should be =
needed to
> be done explicitly.
> Also, after the crash, the operator is restored to the checkpoint =
which had
> happened after beginWindow but before endWindow. In this case the
> beginWindow() call will be skipped when the operator restarts after
> recovery.
> It is easy to wrongly assume (which I did) that checkpoints always =
happen
> "after endWindow" (in which case, the variable can be made transient).
>=20
> I also had a concern regarding *sliding application windows*. It is =
not
> clear as to how the beginWindow and endWindow calls would be =
structured. In
> addition to the application window size, we would also need some =
*delta*
> which is the amount by which the window would "slide" by. How is this
> specified? Is this per tuple? Also how would checkpointing and =
recovery act
> in such a case?
>=20
> Thanks.
> Bhupesh
>=20
> On Mon, Nov 9, 2015 at 12:39 PM, Milind Barve <milindb@gmail.com> =
wrote:
>=20
>> You might want to initialize it in beginwidow() as well. Should keep =
it non
>> transient so for recovery.
>> On Nov 9, 2015 12:34 PM, "Amol Kekre" <amol@datatorrent.com> wrote:
>>=20
>>> Bhupesh.
>>> I would recommend reseting the variable in endWindow call and =
leaving it
>>> non-transient. Usually an empty variable should not add too much to
>>> serialization.
>>>=20
>>> Thks,
>>> Amol
>>>=20
>>>=20
>>> On Sun, Nov 8, 2015 at 10:57 PM, Bhupesh Chawda =
<bhupesh@datatorrent.com
>>>=20
>>> wrote:
>>>=20
>>>> Hi All,
>>>>=20
>>>> I have a question regarding maintaining state of a variable within =
a
>>> single
>>>> application window.
>>>> If I have a variable which is just needed for a single application
>> window
>>>> (just between beginWindow and endWindow calls), do I need to =
serialize
>>> and
>>>> checkpoint it? or can I make it transient?
>>>>=20
>>>> I understand that in case of checkpoint windows being aligned with =
the
>>>> application windows, we can do so. Since in this case, the variable
>> will
>>>> start afresh immediately after the checkpoint. However in case of
>>>> checkpoints happening in between application windows, this might =
not be
>>> the
>>>> case.
>>>>=20
>>>> In the second case, is there any mechanism using which I can
>>> conditionally
>>>> make it transient and non-transient based on when the checkpoints
>> happen
>>>> with respect to the application windows?
>>>>=20
>>>> Thanks.
>>>> Bhupesh
>>>>=20
>>>=20
>>=20


--Apple-Mail=_A7870865-0D08-4E30-99ED-3416EF5ED5A1--