Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
MIME-Version: 1.0
Sender: niels@basj.es
In-Reply-To: 
 <CANC1h_v_HL3tOkUhZ9c59HGoXksZCK5dM+HOZrWhufEQmQy=JQ@mail.gmail.com>
References: 
 <CADoiZqr4e4fmC7u_WyLNAXbakucqQcRsjnzdn5Fj5LM9ZuiX9Q@mail.gmail.com>
	<CANC1h_v_HL3tOkUhZ9c59HGoXksZCK5dM+HOZrWhufEQmQy=JQ@mail.gmail.com>
Date: Fri, 27 Nov 2015 14:19:47 +0100
Message-ID: 
 <CADoiZqoZc0Y_rXgbT3kNuZaWpZ=Hpxrp1WfLt=DhZr6igGaCyQ@mail.gmail.com>
Subject: Re: Cleanup of OperatorStates?
From: Niels Basjes <Niels@basjes.nl>
To: user@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1148eb74a9ed8f0525858b61

--001a1148eb74a9ed8f0525858b61
Content-Type: text/plain; charset=UTF-8

Hi,

Thanks for the explanation.
I have clickstream data arriving in realtime and I need to assign the
visitId and stream it out again (with the visitId now begin part of the
record) into Kafka with the lowest possible latency.
Although the Window feature allows me to group and close the visit on a
timeout/expire (as shown to me by Aljoscha in a separate email) it does
make a 'window'.

So (as requested) I created a ticket for such a feature:
https://issues.apache.org/jira/browse/FLINK-3089

Niels


On Fri, Nov 27, 2015 at 11:51 AM, Stephan Ewen <sewen@apache.org> wrote:

> Hi Niels!
>
> Currently, state is released by setting the value for the key to null. If
> you are tracking web sessions, you can try and send a "end of session"
> element that sets the value to null.
>
> To be on the safe side, you probably want state that is automatically
> purged after a while. I would look into using Windows for that. The
> triggers there are flexible so you can schedule both actions on elements
> plus cleanup after a certain time delay (clock time or event time).
>
> The question about "state expiry" has come a few times. People seem to
> like working on state directly, but it should clean up automatically.
>
> Can you see if your use case fits onto windows, otherwise open a ticket
> for state expiry?
>
> Greetings,
> Stephan
>
>
> On Thu, Nov 26, 2015 at 10:42 PM, Niels Basjes <Niels@basjes.nl> wrote:
>
>> Hi,
>>
>> I'm working on a streaming application that ingests clickstream data.
>> In a specific part of the flow I need to retain a little bit of state per
>> visitor (i.e. keyBy(sessionid) )
>>
>> So I'm using the Key/Value state interface (i.e. OperatorState<MyRecord>)
>> in a map function.
>>
>> Now in my application I expect to get a huge number of sessions per day.
>> Since these sessionids are 'random' and become unused after the visitor
>> leaves the website over time the system will have seen millions of those
>> sessionids.
>>
>> So I was wondering: how are these OperatorStates cleaned?
>>
>>
>> --
>> Best regards / Met vriendelijke groeten,
>>
>> Niels Basjes
>>
>
>


-- 
Best regards / Met vriendelijke groeten,

Niels Basjes

--001a1148eb74a9ed8f0525858b61
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>Thanks for the explanation.</div><d=
iv>I have clickstream data arriving in realtime and I need to assign the vi=
sitId and stream it out again (with the visitId now begin part of the recor=
d) into Kafka with the lowest possible latency.</div><div>Although the Wind=
ow feature allows me to group and close the visit on a timeout/expire (as s=
hown to me by=C2=A0Aljoscha in a separate email) it does make a &#39;window=
&#39;.<br></div><div><br></div><div>So (as requested) I created a ticket fo=
r such a feature:</div><div><a href=3D"https://issues.apache.org/jira/brows=
e/FLINK-3089">https://issues.apache.org/jira/browse/FLINK-3089</a><br></div=
><div><br></div><div>Niels=C2=A0</div><div><br></div><div><br></div><div><b=
r></div><div><br></div><div><br></div></div><div class=3D"gmail_extra"><br>=
<div class=3D"gmail_quote">On Fri, Nov 27, 2015 at 11:51 AM, Stephan Ewen <=
span dir=3D"ltr">&lt;<a href=3D"mailto:sewen@apache.org" target=3D"_blank">=
sewen@apache.org</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote"=
 style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><d=
iv dir=3D"ltr">Hi Niels!<div><br></div><div>Currently, state is released by=
 setting the value for the key to null. If you are tracking web sessions, y=
ou can try and send a &quot;end of session&quot; element that sets the valu=
e to null.</div><div><br></div><div>To be on the safe side, you probably wa=
nt state that is automatically purged after a while. I would look into usin=
g Windows for that. The triggers there are flexible so you can schedule bot=
h actions on elements plus cleanup after a certain time delay (clock time o=
r event time).</div><div><br></div><div>The question about &quot;state expi=
ry&quot; has come a few times. People seem to like working on state directl=
y, but it should clean up automatically.<br></div><div><br></div><div>Can y=
ou see if your use case fits onto windows, otherwise open a ticket for stat=
e expiry?</div><div><br></div><div>Greetings,</div><div>Stephan</div><div><=
br></div></div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_=
extra"><br><div class=3D"gmail_quote">On Thu, Nov 26, 2015 at 10:42 PM, Nie=
ls Basjes <span dir=3D"ltr">&lt;<a href=3D"mailto:Niels@basjes.nl" target=
=3D"_blank">Niels@basjes.nl</a>&gt;</span> wrote:<br><blockquote class=3D"g=
mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l=
eft:1ex"><div dir=3D"ltr">Hi,<div><br></div><div>I&#39;m working on a strea=
ming application that ingests clickstream data.</div><div>In a specific par=
t of the flow I need to retain a little bit of state per visitor (i.e. keyB=
y(sessionid) )</div><div><br></div><div>So I&#39;m using the Key/Value stat=
e interface (i.e.=C2=A0<span style=3D"color:inherit;font-family:Menlo,&#39;=
Lucida Console&#39;,monospace;font-size:inherit;white-space:pre-wrap;line-h=
eight:1.42857;background-color:transparent">OperatorState</span><span style=
=3D"color:inherit;font-family:Menlo,&#39;Lucida Console&#39;,monospace;font=
-size:inherit;white-space:pre-wrap;line-height:1.42857;font-weight:bold;bac=
kground-color:transparent">&lt;</span><span style=3D"color:inherit;font-fam=
ily:Menlo,&#39;Lucida Console&#39;,monospace;font-size:inherit;white-space:=
pre-wrap;line-height:1.42857;background-color:transparent">MyRecord</span><=
span style=3D"color:inherit;font-family:Menlo,&#39;Lucida Console&#39;,mono=
space;font-size:inherit;white-space:pre-wrap;line-height:1.42857;font-weigh=
t:bold;background-color:transparent">&gt;</span>) in a map function.</div><=
div><br></div><div>Now in my application I expect to get a huge number of s=
essions per day.</div><div>Since these sessionids are &#39;random&#39; and =
become unused after the visitor leaves the website over time the system wil=
l have seen millions of those sessionids.</div><div><br></div><div>So I was=
 wondering: how are these OperatorStates cleaned?</div><span><font color=3D=
"#888888"><div><br></div><div><div><br></div>-- <br><div>Best regards / Met=
 vriendelijke groeten,<br><br>Niels Basjes</div>
</div></font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
<div class=3D"gmail_signature">Best regards / Met vriendelijke groeten,<br>=
<br>Niels Basjes</div>
</div>

--001a1148eb74a9ed8f0525858b61--