flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wang, Chao" <chaow...@wustl.edu>
Subject Re: Measure the memory consumption of a job at runtime
Date Fri, 09 Feb 2018 16:58:53 GMT
That helps a lot! Thanks, Kedar!
-Chao

From: kedar mhaswade <kedar.mhaswade@gmail.com>
Date: Thursday, February 8, 2018 at 6:37 PM
To: "Wang, Chao" <chaowang@wustl.edu>
Cc: "user@flink.apache.org" <user@flink.apache.org>
Subject: Re: Measure the memory consumption of a job at runtime

I was also able to attach the JMX (jconsole/VisualVM) console to any container (task manager)
or job manager with these options in your flink-conf.yaml:
env.java.opts: -Djava.rmi.server.hostname=localhost -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=50001
-Dcom.sun.management.jmxremote.rmi.port=50001 -Dcom.sun.management.jmxremote.ssl=false  -Dcom.sun.management.jmxremote.authenticate=false

That way it shows me how the memory is being allocated and collected. Hope that helps:

[Inline image 1]

On Thu, Feb 8, 2018 at 1:55 PM, Wang, Chao <chaowang@wustl.edu<mailto:chaowang@wustl.edu>>
wrote:
Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my Java CEPs. And I’m
interested in measuring the memory overhead of JNI. Though I’m not familiar with how Flink
and the underlying JVM manage memory..

Chao

From: kedar mhaswade <kedar.mhaswade@gmail.com<mailto:kedar.mhaswade@gmail.com>>
Date: Thursday, February 8, 2018 at 3:22 PM
To: "Wang, Chao" <chaowang@wustl.edu<mailto:chaowang@wustl.edu>>
Cc: "user@flink.apache.org<mailto:user@flink.apache.org>" <user@flink.apache.org<mailto:user@flink.apache.org>>
Subject: Re: Measure the memory consumption of a job at runtime

There appears to be some  some support on the Task Manager tab of the Flink dashboard. The
one capability that is missing is to take the thread dump (ala Spark UI).
(Is there an issue/feature request for this?)

[Inline image 2]

Is that what you are looking for?


On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao <chaowang@wustl.edu<mailto:chaowang@wustl.edu>>
wrote:
Hi,

I would like to measure the memory consumption of a job at runtime. I came across some discussion
(here: https://stackoverflow.com/questions/35315522/flink-memory-usage ), and it seems that
it’s not possible two years ago. Is it possible in the current status, and if yes, how to
do it?

Thank you,

Chao


Mime
View raw message