flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwenhael Pasquiers <gwenhael.pasqui...@ericsson.com>
Subject A way to purge an empty session
Date Fri, 23 Jun 2017 16:46:06 GMT
Hello,

This may be premature optimization for memory usage but here is my question :

I have to do an app that will have to monitor sessions of (millions of) users. I don’t know
when the session starts nor ends, nor a reasonable maximum duration.

I want to have a maximum duration (timeout) of 24H.

However I’d like to be able to PURGE sessions that ended as soon as possible to free memory.

I use a Trigger to trig my WindowFunction for each input EVENT. It will (when relevant) output
a SESSION (with start and end timestamps). I use the Evictor in order to remove the EVENT
used to build the SESSION  (they have a Boolean field “removable” set to true from the
WindowFunction so that the Evictor knows it can remove them in the evictAfter method)… That
way at least I can clean the content of the windows.
However From what I’m seeing it looks like the window instance will still stay alive (even
if empty) until it reaches its maximum duration (24 hours) even if the session it represents
lasted 2 minutes: at the end of the day I might have millions of sessions in memory when in
reality only thousands are really alive at a given time. That might also really slow down
the backups and restores of the application if it needs to store millions of empty windows.

I’m aware that my need looks like the session windows. But the session window works mainly
by merging windows that overlap within a session gap. My session gap can be multiple hours
long, so I’m afraid that it would not help me…

So my question is : is there a way to inform the “Trigger” that the windows has no more
elements and that it can be PURGED. Or a way for a WindowFunction to “kill” the window
it’s being applied on ? Of course my window might be re-created if new events arrive later
for the same key.

My other option is to simply use a flatmap operator that will hold an HashMap of sessions,
that way I might be able to clean it up when I close my sessions, but I think it would be
prettier to rely on Flink’s Windows ;-)

Thanks in advance,
Mime
View raw message