flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Memory growth from TimeWindows
Date Mon, 18 May 2020 11:29:15 GMT
On 15.05.20 15:17, Slotterback, Chris wrote:
> My understanding is that while all these windows build their memory state, I can expect
heap memory to grow for the 24 hour length of the SlidingEventTimeWindow, and then start to
flatten as the t-24hr window frames expire and release back to the JVM. What is actually happening
is when a constant data source feeds the stream, the heap memory profile grows linearly past
the 24 hour mark. Could this be a result of a misunderstanding of how the window’s memory
states are kept, or is my assumption correct, and it is more likely I have a leak somewhere?

Will memory keep growing indefinitely? That would indicate a bug? What 
sort of lateness/watermark settings do you have? What window function do 
you use? ProcessWindowFunction, or sth that aggregates?

Side note: with sliding windows of 24h/5min you will have a "write 
amplification" of 24*60/5=288, each record will be in 288 windows, which 
will each be kept in separate state?


View raw message