flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-10066) Keep only archived version of previous executions
Date Mon, 06 Aug 2018 08:28:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-10066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569868#comment-16569868

ASF GitHub Bot commented on FLINK-10066:

StefanRRichter commented on issue #6500: [FLINK-10066] Keep only archived version of previous
URL: https://github.com/apache/flink/pull/6500#issuecomment-410629207
   If yes, I would also consider backporting to the 1.5 and 1.4 branch.

This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:

> Keep only archived version of previous executions
> -------------------------------------------------
>                 Key: FLINK-10066
>                 URL: https://issues.apache.org/jira/browse/FLINK-10066
>             Project: Flink
>          Issue Type: Improvement
>          Components: JobManager
>    Affects Versions: 1.4.3, 1.5.2, 1.6.0
>            Reporter: Stefan Richter
>            Assignee: Stefan Richter
>            Priority: Major
>              Labels: pull-request-available
> Currently, the execution vertex stores a limited amount of previous executions in a bounded
list. This happens primarily for archiving purposes and to remember previous locations and
allocation ids. We remember the whole execution to eventually convert it into an archived
> This seems unnecessary and dangerous as we have observed that this strategy is prone
to memory leaks in the job manager. With a very high vertex count or parallelism, remembering
complete executions can become very memory intensive. Instead I suggest to eagerly transform
the executions into the archived version before adding them to the list, i.e. only the archived
version is ever still referenced after the execution becomes obsolete. This gives better control
over which information about the execution should really be kept in memory.

This message was sent by Atlassian JIRA

View raw message