beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ben Chambers (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-696) Side-Inputs non-deterministic with merging main-input windows
Date Fri, 30 Sep 2016 22:58:21 GMT
Ben Chambers created BEAM-696:
---------------------------------

             Summary: Side-Inputs non-deterministic with merging main-input windows
                 Key: BEAM-696
                 URL: https://issues.apache.org/jira/browse/BEAM-696
             Project: Beam
          Issue Type: Bug
          Components: beam-model
            Reporter: Ben Chambers
            Assignee: Frances Perry


Side-Inputs are non-deterministic for several reasons:
1. Because they depend on triggering of the side-input (this is acceptable because triggers
are by their nature non-deterministic).
2. They depend on the current state of the main-input window in order to lookup the side-input.
This means that with merging
3. Any runner optimizations that affect when the side-input is looked up may cause problems
with either or both of these.

This issue focuses on #2 -- the non-determinism of side-inputs that execute within a Merging
WindowFn.

Possible solution would be to defer running anything that looks up the side-input until we
need to extract an output, and using the main-window at that point. Specifically, if the main-window
is a MergingWindowFn, don't execute any kind of pre-combine, instead buffer all the inputs
and combine later.

This could still run into some non-determinism if there are triggers controlling when we extract
output.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message