beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-654) When and how can merging windows "shrink" or "grow"?
Date Tue, 20 Sep 2016 22:49:20 GMT
Kenneth Knowles created BEAM-654:
------------------------------------

             Summary: When and how can merging windows "shrink" or "grow"?
                 Key: BEAM-654
                 URL: https://issues.apache.org/jira/browse/BEAM-654
             Project: Beam
          Issue Type: New Feature
          Components: beam-model
            Reporter: Kenneth Knowles
            Assignee: Kenneth Knowles


The primary example of a merging window today is {{Sessions}} by gap duration, in which the
merged window is the interval enclosure / span of the windows being merged.

However, another reasonable abstract use case is a session identified by id with an explicit
end event. We might consider that the session ends with no gap duration after the end event.
In this case, the merged window may be smaller than the enclosure of the sub-windows. Sometimes
this has been referred to as "merging shrinks the window".

Perhaps the only requirement is that the merged window contains the timestamps of the data
therein, but we should document this clearly. The current spec is ["Does whatever merging
is necessary"|https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/windowing/WindowFn.java#L106]

There are repercussions for triggers, some documented in the [trigger design doc|https://s.apache.org/beam-triggers]:
With nonzero allowed lateness, {{Sessions}} by gap duration can switch a trigger from ON_TIME
or LATE behavior back to speculative behavior, or cause another ON_TIME firing. Conversely,
sessions with abrupt termination/shrinking may have that behavior _as well as_ an ON_TIME
and subsequent LATE firings due only to the merging (this already works properly).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message