beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-2571) Flink ValidatesRunner failing CombineTest.testSlidingWindowsCombineWithContext
Date Tue, 11 Jul 2017 07:33:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16081817#comment-16081817
] 

Aljoscha Krettek commented on BEAM-2571:
----------------------------------------

What's the problem in Dataflow?

In the real world this virtually doesn't occur, IMHO. The reason we're catching this here
is because of the side inputs and the holding back of the main input. Things go roughly like
this:
 - main-input elements arrive, {{DoFnOperator}} puts them in state and holds the input watermark
back according to their timestamp
 - input watermark already goes to +Inf at this point
 - the first side-input element arrives, we notice that we can process one of the pushed-pack
elements, the watermark is still "tightly held" by the other pushed-back elements, that's
why it is exactly at the window end for some windows

> Flink ValidatesRunner failing CombineTest.testSlidingWindowsCombineWithContext
> ------------------------------------------------------------------------------
>
>                 Key: BEAM-2571
>                 URL: https://issues.apache.org/jira/browse/BEAM-2571
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>            Reporter: Kenneth Knowles
>            Assignee: Aljoscha Krettek
>             Fix For: 2.1.0
>
>
> This appears to have been caused by https://github.com/apache/beam/pull/3429 which fixes
a couple errors in how trigger timers were processed / final panes labeled.
> I am investigating, considering roll back vs forward fix. Since it is an esoteric use
case where I would advise users to use a stateful DoFn instead, I think the bug fixed probably
outweighs the bug introduced. I would like to fix for 2.1.0 but will report back soon.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message