flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-5972) Don't allow shrinking merging windows
Date Mon, 06 Mar 2017 16:09:32 GMT
Aljoscha Krettek created FLINK-5972:

             Summary: Don't allow shrinking merging windows
                 Key: FLINK-5972
                 URL: https://issues.apache.org/jira/browse/FLINK-5972
             Project: Flink
          Issue Type: Bug
          Components: DataStream API
    Affects Versions: 1.2.0, 1.1.0, 1.3.0
            Reporter: Aljoscha Krettek
            Priority: Blocker
             Fix For: 1.3.0

A misbehaving {{MergingWindowAssigner}} can cause a merge that results in a window that is
smaller than the span of all the merged windows. This, in itself is not problematic. It becomes
problematic when the end timestamp of a window that was not late before merging is now earlier
than the watermark (the timestamp is smaller than the watermark).

There are two choices:
 - immediately process the window
 - drop the window

processing the window will lead to late data downstream.

The current behaviour is to silently drop the window but that logic has a bug: we only remove
the dropped window from the {{MergingWindowSet}} but we don't properly clean up state and
timers that the window still (possibly) has. We should fix this bug in the process of resolving
this issue.

We should either just fix the bug and still silently drop windows or add a check and throw
an exception when the end timestamp falls below the watermark.

This message was sent by Atlassian JIRA

View raw message