beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3151) AfterProcessingTime trigger doesn't create expected file panes
Date Tue, 07 Nov 2017 19:10:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16242663#comment-16242663
] 

Kenneth Knowles commented on BEAM-3151:
---------------------------------------

Because you have specified {{.withNumShards(1)}} there is another grouping within the TextIO
transform that will buffer and trigger, so all elements will be on a single key and governed
by an internal-only trigger that synchronizes on all upstream processing time outputs. The
resulting panes and first/last are nondeterministic - they could be all in a non-final output,
in multiple non-final outputs, all in a final output, or split across non-final and final
outputs.

When I set up a simulation of that, I get a non-final pane with {{"A", 2}} and a final output
with {{"B", 2"}} and {{"C", 2}}. But again, the configuration allows many variations, as long
as the sharded file contains all the contents.

> AfterProcessingTime trigger doesn't create expected file panes
> --------------------------------------------------------------
>
>                 Key: BEAM-3151
>                 URL: https://issues.apache.org/jira/browse/BEAM-3151
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>    Affects Versions: 2.0.0, 2.1.0
>            Reporter: Pawel Bartoszek
>            Assignee: Kenneth Knowles
>
> I am seeing some weird behaviour with Beam 2.0.0 about file panes created when using
 AfterProcessingTime trigger. When I switch to Beam 2.1.0 I am getting different behaviour,
though is not what I would expect as well. In the test I included the expected results and
actual results produced by Beam 2.0.0 and Beam 2.1.0. I am using Direct Runner
> The test can be found at:
> [https://gist.github.com/pbartoszek/b9e7d96c75cff52076125ef47d3f69f9|https://gist.github.com/pbartoszek/b9e7d96c75cff52076125ef47d3f69f9]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message