flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: Off heap memory issue
Date Wed, 30 Aug 2017 08:27:22 GMT
Hi Javier,

I'm not aware of such issues with Flink, but if you could give us some more
details on your setup, I might get some more ideas on what to look for.

are you using the RocksDBStateBackend? (RocksDB is doing some JNI
allocations, that could potentially leak memory)
Also, are you passing any special garbage collector options? (Maybe some
classes are not unloaded)
Are you using anything else that is special (such as protobuf or avro
formats, or any other big library)?

Regards,
Robert



On Mon, Aug 28, 2017 at 5:04 PM, Javier Lopez <javier.lopez@zalando.de>
wrote:

> Hi all,
>
> we are starting a lot of Flink jobs (streaming), and after we have started
> 200 or more jobs we see that the non-heap memory in the taskmanagers
> increases a lot, to the point of killing the instances. We found out that
> every time we start a new job, the committed non-heap memory increases by 5
> to 10MB. Is this an expected behavior? Are there ways to prevent this?
>

Mime
View raw message