beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-1396) GABWVOBDoFn expects grouped values to be ordered by their timestamp but there is no such guarantee
Date Sun, 05 Feb 2017 17:48:42 GMT
Amit Sela created BEAM-1396:
-------------------------------

             Summary: GABWVOBDoFn expects grouped values to be ordered by their timestamp
but there is no such guarantee
                 Key: BEAM-1396
                 URL: https://issues.apache.org/jira/browse/BEAM-1396
             Project: Beam
          Issue Type: Bug
          Components: sdk-java-core
            Reporter: Amit Sela
            Assignee: Kenneth Knowles


GABWVOBDoFn relies on the grouped values to be ordered by their timestamp but nothing in the
SDK guarantees this: 
https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/GroupAlsoByWindowsViaOutputBufferDoFn.java#L86

If such a chunk of timestamped values will be processed out-of-order I assume we'd end up
with an {{IllegalStateException}} thrown here:
https://github.com/apache/beam/blob/master/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java#L191

I suggest we go ahead and add sorting before processing the bundle in chunks - this might
prove expensive in extreme cases where a very large bundle with very few keys is processed,
but it seems that timestamp order is necessary.
As for runners who provide order guarantee, since GABW is optional I don't see an issue here,
though [~dhalperi@google.com] suggested we add a "shouldSort" flag.

Also, probably worth creating a test for this, though it would prove difficult since we would
have to preset the order which is the problem to begin with :-)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message