beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (BEAM-145) OutputTimeFn#assignOutputTime overrides WindowFn#getOutputTime in unfortunate ways
Date Thu, 23 Mar 2017 05:12:41 GMT

     [ https://issues.apache.org/jira/browse/BEAM-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kenneth Knowles resolved BEAM-145.
----------------------------------
       Resolution: Duplicate
    Fix Version/s: First stable release

The appropriate status would be "Redundant" in that the existing Runner API proto combined
with our intent to use it and imminent use of it make this irrelevant now.

> OutputTimeFn#assignOutputTime overrides WindowFn#getOutputTime in unfortunate ways
> ----------------------------------------------------------------------------------
>
>                 Key: BEAM-145
>                 URL: https://issues.apache.org/jira/browse/BEAM-145
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-core
>            Reporter: Kenneth Knowles
>            Assignee: Kenneth Knowles
>            Priority: Minor
>              Labels: windowing
>             Fix For: First stable release
>
>
> Today the {{OutputTimeFn}} includes {{#assignOutputTime}}, {{#combine}}, and {{#merge}}.
Together these express the grouping of timestamps, analogous to the grouping of values in
a GBK / Combine, in a canonical way.
> The default {{OutputTimeFn}} is provided by the {{WindowFn}}. In particular, {{SlidingWindows}}
provides an {{OutputTimeFn}} that shifts input timestamps later to avoid watermark stuckness
and then takes the minimum to compute the output timestamp.
> The SDK additionally provides instance for "min", "max" and "end of window" output timestamps.
> Unfortunately,  if one overrides the {{OutputTimeFn}} to one of these, the shifting done
by {{SlidingWindows}} is lost.
> This is actually only a minor problem for now, since "min" is the default, "end of window"
is unaffected, and "max" has only esoteric uses.The fix is easy:
> This is interrelated with another suggested change:  Since there are only three common
{{OutputTimeFn}} instances, and it is a high bandwidth API, it does not seem worthwhile to
leave it in userland. So it is proposed to reduce it to an enum, which would leave only the
{{WindowFn}} as a userland place for timestamp adjustments. (requiring special casing for
end-of-window, since it cannot be implemented without owning {{#assignOutputTime}})



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message