flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Theo Diefenthal <theo.diefent...@scoop-software.de>
Subject Re: Help to Understand cutoff memory
Date Tue, 10 Dec 2019 22:30:40 GMT
Hi Lu, 

I found this talk on last Flink Forward in Berlin very helpful in order to understand JVM
RAM and cutoff memory [1]. Maybe it helps you understand that stuff better. 
In my experiences on YARN, the author was totally correct. I was able to reproduce that by
assigning something about 12GB for taskmanager memory, setting cutoff to 0.15 and let network
stay on 0.1 , I got something round about 8-9GB RAM for my real taskmanager JVM memory. 

Best regards 
Theo 

[1] [ https://www.youtube.com/watch?v=aq1Whga-RJ4 | https://www.youtube.com/watch?v=aq1Whga-RJ4
] 


Von: "Lu Niu" <qqibrow@gmail.com> 
An: "user" <user@flink.apache.org> 
Gesendet: Dienstag, 10. Dezember 2019 22:58:01 
Betreff: Help to Understand cutoff memory 

Hi, flink users 

I have some question regarding memory allocation. According to doc, containerized.heap-cutoff-ratio
means: 

``` 
Percentage of heap space to remove from containers (YARN / Mesos), to compensate for other
JVM memory usage 
``` 
However, I find cutoff memory is actually treated as "part of direct memory": 
[ https://github.com/apache/flink/blob/release-1.9.1/flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/ContaineredTaskManagerParametersTest.java#L67
| https://github.com/apache/flink/blob/release-1.9.1/flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/ContaineredTaskManagerParametersTest.java#L67
] 

The code above shows the max of MaxDirectMemorySize of jvm process is the sum of networkbuffer
+ cutoff. Then, there is no guarantee of a fixed headroom in container memory. In our case
we use rocksDB memory and we found many times DirectMemory is close to maximum, leaving less
headroom in the container for rocksdb. Could someone help on this? thanks 

Best 
Lu 

Mime
View raw message