flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <till.rohrm...@gmail.com>
Subject Re: [jira] [Created] (FLINK-1843) Job History gets cleared too fast
Date Wed, 08 Apr 2015 10:31:02 GMT
This also happens for cluster setups.

On Wed, Apr 8, 2015 at 11:29 AM, Maximilian Michels (JIRA) <jira@apache.org>
wrote:

> Maximilian Michels created FLINK-1843:
> -----------------------------------------
>
>              Summary: Job History gets cleared too fast
>                  Key: FLINK-1843
>                  URL: https://issues.apache.org/jira/browse/FLINK-1843
>              Project: Flink
>           Issue Type: Bug
>           Components: JobManager
>     Affects Versions: 0.9
>             Reporter: Maximilian Michels
>              Fix For: 0.9
>
>
> As per FLINK-1442, the JobManager stores the archived ExecutionGraph
> behind a SoftReference. At least for local setups, this mechanism doesn't
> seem to work properly. There are two issues:
>
> - The history gets cleared too fast
> - The history gets cleared in a non-sequential fashion, i.e. arbitrary old
> ExecutionGraph are discarded
>
> To solve these problems we might
>
> - Store the least recent ExecutionGraph behind a SoftReference
> - Store the most recent ExecutionGraphs without a SoftReference
>
> That way, we can save memory but have the latest history available to the
> user. We might introduce a configuration variable where the user can
> specify the number of ExecutionGraphs that should be held in memory. The
> remaining can be stored behind a SoftReference.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message