flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kien Truong <duckientru...@gmail.com>
Subject Re: Measure the memory consumption of a job at runtime
Date Tue, 13 Feb 2018 10:05:54 GMT
Hi,

Just want to share a tip, for job with rocksdb backend, you can use 
LD_LIBRARY_PATH to load jemalloc

|containerized.master.env.LD_LIBRARY_PATH: "/usr/lib/native"|


Then use jemalloc heap profiler to track native memory usage by rocksdb.

https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Heap-Profiling


In my experience, jemalloc helps a lot at reducing memory fragmentation 
when using rocksdb state backend.


Best regards,

Kien


On 2/9/2018 11:58 PM, Wang, Chao wrote:
>
> That helps a lot! Thanks, Kedar!
>
> -Chao
>
> *From: *kedar mhaswade <kedar.mhaswade@gmail.com>
> *Date: *Thursday, February 8, 2018 at 6:37 PM
> *To: *"Wang, Chao" <chaowang@wustl.edu>
> *Cc: *"user@flink.apache.org" <user@flink.apache.org>
> *Subject: *Re: Measure the memory consumption of a job at runtime
>
> I was also able to attach the JMX (jconsole/VisualVM) console to any 
> container (task manager) or job manager with these options in your 
> flink-conf.yaml:
>
> env.java.opts: -Djava.rmi.server.hostname=localhost 
> -Dcom.sun.management.jmxremote=true 
> -Dcom.sun.management.jmxremote.port=50001 
> -Dcom.sun.management.jmxremote.rmi.port=50001 
> -Dcom.sun.management.jmxremote.ssl=false 
> -Dcom.sun.management.jmxremote.authenticate=false
>
> That way it shows me how the memory is being allocated and collected. 
> Hope that helps:
>
> Inline image 1
>
> On Thu, Feb 8, 2018 at 1:55 PM, Wang, Chao 
> <chaowang@wustl.edu<mailto:chaowang@wustl.edu>> wrote:
>
>     Thanks. I’ve been plugging C/C++ libraries (e.g., libgcrypt) in my
>     Java CEPs. And I’m interested in measuring the memory overhead of
>     JNI. Though I’m not familiar with how Flink and the underlying JVM
>     manage memory..
>
>     Chao
>
>     *From: *kedar mhaswade
>     <kedar.mhaswade@gmail.com<mailto:kedar.mhaswade@gmail.com>>
>     *Date: *Thursday, February 8, 2018 at 3:22 PM
>     *To: *"Wang, Chao" <chaowang@wustl.edu<mailto:chaowang@wustl.edu>>
>     *Cc: *"user@flink.apache.org<mailto:user@flink.apache.org>"
>     <user@flink.apache.org<mailto:user@flink.apache.org>>
>     *Subject: *Re: Measure the memory consumption of a job at runtime
>
>     There appears to be some  some support on the Task Manager tab of
>     the Flink dashboard. The one capability that is missing is to take
>     the thread dump (ala Spark UI).
>
>     (Is there an issue/feature request for this?)
>
>     Inline image 2
>
>     Is that what you are looking for?
>
>     On Thu, Feb 8, 2018 at 12:33 PM, Wang, Chao
>     <chaowang@wustl.edu<mailto:chaowang@wustl.edu>> wrote:
>
>         Hi,
>
>         I would like to measure the memory consumption of a job at
>         runtime. I came across some discussion (here:
>         https://stackoverflow.com/questions/35315522/flink-memory-usage),
>         and it seems that it’s not possible two years ago. Is it
>         possible in the current status, and if yes, how to do it?
>
>         Thank you,
>
>         Chao
>

Mime
View raw message