flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hao Sun <ha...@zendesk.com>
Subject Re: org.apache.flink.runtime.io.network.NetworkEnvironment causing memory leak?
Date Thu, 16 Nov 2017 15:44:45 GMT
Thanks a lot! This is very helpful.
In addition to your comments, what are the items retained by
NetworkEnvironment? They grew seems like indefinitely, do they ever reduce?

I think there is a GC issue because my task manager is killed somehow after
a job run. The duration correlates to the volume of Kafka topics. More
volume TM dies quickly. Do you have any tips to debug it?

On Thu, Nov 16, 2017, 01:35 Stefan Richter <s.richter@data-artisans.com>
wrote:

> Hi,
>
> I cannot spot anything that indicates a leak from your screenshots. Maybe
> you misinterpret the numbers? In your heap dump, there is only a single
> instance of org.apache.flink.runtime.io.network.NetworkEnvironment and it
> retains about 400,000,000 bytes from being GCed because it holds references
> to the network buffers. This is perfectly normal because this the buffer
> pool is part of this object, and for as long as it lives, the referenced
> buffers should not be GCed and the current size of all your buffers is
> around 400 million bytes.
>
> Your heap space is also not growing without bounds, but always goes down
> after a GC was performed. Looks fine to me.
>
> Last, I think the number of G1_Young_Generation is a counter of how many
> gc cycles have been performed and the time is a sum. So naturally, those
> values would always increase.
>
> Best,
> Stefan
>
> > Am 15.11.2017 um 18:35 schrieb Hao Sun <hasun@zendesk.com>:
> >
> > Hi team, I am looking at some memory/GC issues for my flink setup. I am
> running flink 1.3.2 in docker for my development environment. Using
> Kubernetes for production.
> > I see instances of org.apache.flink.runtime.io.network.NetworkEnvironment
> are increasing dramatically and not GC-ed very well for my application.
> > My simple app collects Kafka events and transforms the information and
> logs the results out.
> >
> > Is this expected? I am new to Java memory analysis not sure what is
> actually wrong.
> >
> > <image.png>
> > <image.png>
> > <image.png>
> > <image.png>
>
>

Mime
View raw message