flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [flink-web] MarkSfik commented on a change in pull request #373: Add blog post: Memory Management improvements for Flink’s JobManager in Apache Flink 1.11
Date Mon, 31 Aug 2020 07:15:22 GMT

MarkSfik commented on a change in pull request #373:
URL: https://github.com/apache/flink-web/pull/373#discussion_r479940629



##########
File path: _posts/2020-09-01-flink-1.11-memory-management-improvements.md
##########
@@ -0,0 +1,66 @@
+---
+layout: post
+title: "Memory Management improvements for Flink’s JobManager in Apache Flink 1.11"
+date: 2020-09-01T15:30:00.000Z
+authors:
+- Andrey:
+  name: "Andrey Zagrebin"
+categories: news
+excerpt: In a previous blog post focused on the memory model of the TaskManagers and how
it was improved with the Apache Flink 1.10 release. This blog post addresses the same topic
but for the JobManager instead.
+---
+
+Apache Flink 1.11 comes with significant changes to the memory model of Flink’s JobManager
and configuration options for your Flink clusters. These recently-introduced changes make
Flink adaptable to all kinds of deployment environments (e.g. Kubernetes, Yarn, Mesos), providing
better control over its memory consumption.
+
+The [previous blog post]({{ site.baseurl }}/news/2020/04/21/memory-management-improvements-flink-1.10.html),
focused on the memory model of the TaskManagers and how it was improved with the Apache Flink
1.10 release. This blog post addresses the same topic but for the JobManager instead. Flink
1.11 unifies the memory model of Flink’s processes. The newly introduced memory model of
the JobManager follows a  similar approach to that of the TaskManagers; it is simpler and
has fewer components and tuning knobs. This post might then seem very similar to our previous
story on Flink’s memory, but aims at providing a complete overview of Flink’s JobManager
memory model as of Flink 1.11. Read on for a full list of updates and changes below!
+
+## Introduction to Flink’s process memory model
+
+Having a clear understanding of Apache Flink’s process memory model allows you to manage
resources for the various workloads more efficiently. The following diagram illustrates the
main memory components of a Flink process:
+
+<center>
+<img src="{{ site.baseurl }}/img/blog/2020-09-01-flink-1.11-memory-management-improvements/total-process-memory-flink-1.11.png"
width="400px" alt="Backpressure sampling:high"/>
+<br/>
+<i><small>Flink: Total Process Memory</small></i>
+</center>
+<br/>
+
+The JobManager process is a JVM process. On a high level, its memory consists of the JVM
Heap and Off-Heap memory. These types of memory are consumed by Flink directly or by JVM for
its specific purposes (i.e. metaspace etc). There are two major memory consumers within the
JobManager process: the framework itself consuming memory for internal data structures, network
communication, etc. and the user code which runs within the JobManager process, e.g. in certain
batch sources or in checkpoint completion callbacks.
+
+<div class="alert alert-info" markdown="1">
+<span class="label label-info" style="display: inline-block"><span class="glyphicon
glyphicon-info-sign" aria-hidden="true"></span> Note</span>
+Please, note that the user code has direct access to all memory types: JVM Heap, Direct and
Native memory. Therefore, Flink cannot really control its allocation and usage.

Review comment:
       ```suggestion
   Please note that the user code has direct access to all memory types: JVM Heap, Direct
and Native memory. Therefore, Flink cannot really control its allocation and usage.
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



Mime
View raw message