flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: Can someone explain memory usage in a flink worker?
Date Thu, 27 Jul 2017 10:21:27 GMT
I also faced annoying problems with Flink memory and TMs killed by the OS
because of OOM.
To limit somehow the memory consumption of TM *on a single job* I do the
following:

Add to flink.yaml

   - env.java.opts: -Dio.netty.recycler.maxCapacity.default=1

Edit taskmanager.sh

   - and change TM_MAX_OFFHEAP_SIZE from 8388607T to 5g


PROBABLY the unexpected memory consumption is causeb by Netty and this
allow a single job to terminate without being killed by the OS.
However, the TM memory continuosly grows one job after the other..it seems
that Flink doesn't free all the memory somehow (but I don't know where).

I hope this helps,
Flavio

On Thu, Jul 27, 2017 at 11:29 AM, Shashwat Rastogi <
shashwat.rastogi@reflektion.com> wrote:

> Hi,
>
> I have a setup of 7 task managers, each with 64GB of physical memory out
> of which I have allocated 35GB as task manager’s heap memory. I am using
> rocksdb as state backend.
>
> I see a lot of anomalies in the reports generated by the Flink-UI vs my
> system metrics. Can someone please help explaining what is happening here.
>
> - Why does the flink-ui shows 35 GB as free memory when the system has
> currently occupied 48.4 GB of memory, which leaves only (62.5-48.4)14.1 GB
> free.
> - Where is the memory used by RocksDb displayed? The machine does not do
> anything except serving as the flink worker, so can I assume that 62.5 GB -
> 35 GB - 14.1 Gb - 523MB = 12.9 GB is the memory used by RocksDb? When I do `*ps
> -ef | grep rocksdb` *I don’t see any process running, is this normal?
> - Also, my system metrics shows that the memory usage keeps on increasing until
> the task-manager itself gets killed
> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Memory-Leak-Flink-RocksDB-td14439.html>
but
> when I see in the Flink-Ui I always see lot of free memory. I am using the
> default configuration so I don’t think flink managed memory occupies
> non-heap memory. I am not able to figure out where does this
> ever-increasing memory consumption is coming from, my guess is this is used
> by RocksDb.
>
> *FLINK-UI*
>
>
>
> *SYSTEM METRICS*
>
>
>
> Thanks in advance.
> Shashwat
>

Mime
View raw message