flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwenhael Pasquiers <gwenhael.pasqui...@ericsson.com>
Subject RE: A way to purge an empty session
Date Mon, 26 Jun 2017 14:22:13 GMT
Thanks !

I didn’t know of this function and indeed it seems to match my needs better than Windows.
And I’ll be able to clear my state once it’s empty (and re-create it when necessary).


From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: lundi 26 juin 2017 12:13
To: Gwenhael Pasquiers <gwenhael.pasquiers@ericsson.com>
Cc: user@flink.apache.org
Subject: Re: A way to purge an empty session

Hi Gwenhael,
have you considered to use a ProcessFunction? With a ProcessFunction you have direct access
to state (that you register yourself) and you can register timers that trigger a callback
function when they expire.
So you can cleanup state when you receive an element or when a timer expires.

Best, Fabian

2017-06-23 18:46 GMT+02:00 Gwenhael Pasquiers <gwenhael.pasquiers@ericsson.com<mailto:gwenhael.pasquiers@ericsson.com>>:

This may be premature optimization for memory usage but here is my question :

I have to do an app that will have to monitor sessions of (millions of) users. I don’t know
when the session starts nor ends, nor a reasonable maximum duration.
I want to have a maximum duration (timeout) of 24H.

However I’d like to be able to PURGE sessions that ended as soon as possible to free memory.

I use a Trigger to trig my WindowFunction for each input EVENT. It will (when relevant) output
a SESSION (with start and end timestamps). I use the Evictor in order to remove the EVENT
used to build the SESSION  (they have a Boolean field “removable” set to true from the
WindowFunction so that the Evictor knows it can remove them in the evictAfter method)… That
way at least I can clean the content of the windows.
However From what I’m seeing it looks like the window instance will still stay alive (even
if empty) until it reaches its maximum duration (24 hours) even if the session it represents
lasted 2 minutes: at the end of the day I might have millions of sessions in memory when in
reality only thousands are really alive at a given time. That might also really slow down
the backups and restores of the application if it needs to store millions of empty windows.

I’m aware that my need looks like the session windows. But the session window works mainly
by merging windows that overlap within a session gap. My session gap can be multiple hours
long, so I’m afraid that it would not help me…

So my question is : is there a way to inform the “Trigger” that the windows has no more
elements and that it can be PURGED. Or a way for a WindowFunction to “kill” the window
it’s being applied on ? Of course my window might be re-created if new events arrive later
for the same key.

My other option is to simply use a flatmap operator that will hold an HashMap of sessions,
that way I might be able to clean it up when I close my sessions, but I think it would be
prettier to rely on Flink’s Windows ;-)

Thanks in advance,

View raw message