beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <>
Subject [jira] [Commented] (BEAM-3151) AfterProcessingTime trigger doesn't create expected file panes
Date Tue, 07 Nov 2017 19:10:00 GMT


Kenneth Knowles commented on BEAM-3151:

Because you have specified {{.withNumShards(1)}} there is another grouping within the TextIO
transform that will buffer and trigger, so all elements will be on a single key and governed
by an internal-only trigger that synchronizes on all upstream processing time outputs. The
resulting panes and first/last are nondeterministic - they could be all in a non-final output,
in multiple non-final outputs, all in a final output, or split across non-final and final

When I set up a simulation of that, I get a non-final pane with {{"A", 2}} and a final output
with {{"B", 2"}} and {{"C", 2}}. But again, the configuration allows many variations, as long
as the sharded file contains all the contents.

> AfterProcessingTime trigger doesn't create expected file panes
> --------------------------------------------------------------
>                 Key: BEAM-3151
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>    Affects Versions: 2.0.0, 2.1.0
>            Reporter: Pawel Bartoszek
>            Assignee: Kenneth Knowles
> I am seeing some weird behaviour with Beam 2.0.0 about file panes created when using
 AfterProcessingTime trigger. When I switch to Beam 2.1.0 I am getting different behaviour,
though is not what I would expect as well. In the test I included the expected results and
actual results produced by Beam 2.0.0 and Beam 2.1.0. I am using Direct Runner
> The test can be found at:
> [|]

This message was sent by Atlassian JIRA

View raw message