flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Richter (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (FLINK-10066) Keep only archived version of previous executions
Date Mon, 13 Aug 2018 18:17:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-10066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Stefan Richter closed FLINK-10066.
    Resolution: Fixed

Merged in:
master: 160dc56fdf
release-1.6: 74323d50b0
release-1.5: 2217c09c88

> Keep only archived version of previous executions
> -------------------------------------------------
>                 Key: FLINK-10066
>                 URL: https://issues.apache.org/jira/browse/FLINK-10066
>             Project: Flink
>          Issue Type: Improvement
>          Components: JobManager
>    Affects Versions: 1.4.3, 1.5.2, 1.6.0
>            Reporter: Stefan Richter
>            Assignee: Stefan Richter
>            Priority: Major
>              Labels: pull-request-available
> Currently, the execution vertex stores a limited amount of previous executions in a bounded
list. This happens primarily for archiving purposes and to remember previous locations and
allocation ids. We remember the whole execution to eventually convert it into an archived
> This seems unnecessary and dangerous as we have observed that this strategy is prone
to memory leaks in the job manager. With a very high vertex count or parallelism, remembering
complete executions can become very memory intensive. Instead I suggest to eagerly transform
the executions into the archived version before adding them to the list, i.e. only the archived
version is ever still referenced after the execution becomes obsolete. This gives better control
over which information about the execution should really be kept in memory.

This message was sent by Atlassian JIRA

View raw message