flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-2317) Stream Jobs with Windows Leave Lingering State
Date Fri, 03 Jul 2015 08:39:04 GMT
Aljoscha Krettek created FLINK-2317:

             Summary: Stream Jobs with Windows Leave Lingering State
                 Key: FLINK-2317
                 URL: https://issues.apache.org/jira/browse/FLINK-2317
             Project: Flink
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 0.9, 0.9.1
            Reporter: Aljoscha Krettek
            Priority: Blocker

The problem is that some components in the windowing system create Threads that are not properly
cleaned up after a Job finished (is canceled). Specifically these are the {{CentralCheck}}
thread of the {{GroupedActiveDiscretizer}} and the {{TimeCheck}} thread in {{TimeTriggerPolicy}}.
These threads have references to other parts of the windowing system, such as the window buffers.
This then means that the complete state of the buffers at the time a job is canceled stays
around indefinitely. Over time this will fill up the memory and make successive windowing
jobs incredibly slow or lead to GC overheat exceptions.

Also, to prevent stuff like this from happening in the future we should put all streaming
Tests on a common {{TestBase}} that verifies that we have no lingering threads after job execution.

This message was sent by Atlassian JIRA

View raw message