flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2004) Memory leack in presence of failed checkpoints in KafkaSource
Date Tue, 12 May 2015 19:57:01 GMT
Stephan Ewen created FLINK-2004:
-----------------------------------

             Summary: Memory leack in presence of failed checkpoints in KafkaSource
                 Key: FLINK-2004
                 URL: https://issues.apache.org/jira/browse/FLINK-2004
             Project: Flink
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 0.9
            Reporter: Stephan Ewen
            Assignee: Robert Metzger
            Priority: Critical
             Fix For: 0.9


Checkpoints that fail never send a commit message to the tasks.

Maintaining a map of all pending checkpoints introduces a memory leak, as entries for failed
checkpoints will never be removed.

Approaches to fix this:
  - The source cleans up entries from older checkpoints once a checkpoint is committed (simple
implementation in a linked hash map)
  - The commit message could include the optional state handle (source needs not maintain
the map)
  - The checkpoint coordinator could send messages for failed checkpoints?





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message