flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wesley Peng <wesley.pe...@googlemail.com>
Subject Re: flink sql row_number() over () OOM
Date Wed, 04 Sep 2019 11:47:55 GMT
Hi

on 2019/9/4 19:30, liu ze wrote:
> I use the row_number() over() function to do topN, the total amount of 
> data is 60,000, and the state is 12G .
> Finally, oom, is there any way to optimize it?

ref: 
https://stackoverflow.com/questions/50812837/flink-taskmanager-out-of-memory-and-memory-configuration

The total amount of required physical and heap memory is quite difficult 
to compute since it strongly depends on your user code, your job's 
topology and which state backend you use.

As a rule of thumb, if you experience OOM and are still using the 
FileSystemStateBackend or the MemoryStateBackend, then you should switch 
to RocksDBStateBackend, because it can gracefully spill to disk if the 
state grows too big.

If you are still experiencing OOM exceptions as you have described, then 
you should check your user code whether it keeps references to state 
objects or generates in some other way large objects which cannot be 
garbage collected. If this is the case, then you should try to refactor 
your code to rely on Flink's state abstraction, because with RocksDB it 
can go out of core.

RocksDB itself needs native memory which adds to Flink's memory 
footprint. This depends on the block cache size, indexes, bloom filters 
and memtables. You can find out more about these things and how to 
configure them here.

Last but not least, you should not activate 
taskmanager.memory.preallocate when running streaming jobs, because 
streaming jobs currently don't use managed memory. Thus, by activating 
preallocation, you would allocate memory for Flink's managed memory 
which is reduces the available heap space.

Mime
View raw message