Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Sat, 15 Apr 2017 06:43:41 +0000 (UTC)
From: "Aljoscha Krettek (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13064174.1492213288000.289202.1492238621535@Atlassian.JIRA>
In-Reply-To: <JIRA.13064174.1492213288000@Atlassian.JIRA>
References: <JIRA.13064174.1492213288000@Atlassian.JIRA> <JIRA.13064174.1492213288977@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-1983) SDF should properly support windowed
 side inputs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Sat, 15 Apr 2017 06:43:47 -0000


    [ https://issues.apache.org/jira/browse/BEAM-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969830#comment-15969830 ] 

Aljoscha Krettek commented on BEAM-1983:
----------------------------------------

This is related to BEAM-1850.

[~jkff] Remember we had the discussion on why this part is a bit strange: https://github.com/apache/beam/blob/5824bb4b5700b5230f569c570d5e8ed2d11cedf2/runners/flink/runner/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L1081-L1081?

> SDF should properly support windowed side inputs
> ------------------------------------------------
>
>                 Key: BEAM-1983
>                 URL: https://issues.apache.org/jira/browse/BEAM-1983
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-apex, runner-dataflow, runner-direct, runner-flink, sdk-java-core
>            Reporter: Eugene Kirpichov
>            Assignee: Eugene Kirpichov
>
> Currently there is no test coverage for Splittable DoFn + windowed side inputs, especially when not all of the side input windows are ready.
> Moreover, current implementation of SDF in the direct runner is definitely wrong: it uses a ParDoEvaluator to run the ProcessFn, and this ParDoEvaluator looks at the wrong windows to decide which windows are ready and which are not: https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/ParDoEvaluator.java#L134 - the WindowedValue in question is a KeyedWorkItem, and they are always in the global window, but the important windows are windows of elements inside this KWI's elementsIterable().
> The Flink implementation is also wrong in the same way.
> This JIRA is to:
> 1) add test coverage for this case
> 2) implement proper support in all runners
> I believe the easiest way to do 2) is to:
> - make SplittableParDo, in case the DoFn has side inputs, pre-explode windows before feeding them into GroupByKeyIntoKeyedWorkItems , so that the resulting KWI's have elements only in a single window
> - tweak runners to look at the proper window, and assert that there's only one window, while evaluating ProcessFn, in case the DoFn uses side inputs


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)