Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Wed, 19 Apr 2017 22:46:04 +0000 (UTC)
From: "Kenneth Knowles (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13036339.1484859709000.62.1492641964615@Atlassian.JIRA>
In-Reply-To: <JIRA.13036339.1484859709000@Atlassian.JIRA>
References: <JIRA.13036339.1484859709000@Atlassian.JIRA> <JIRA.13036339.1484859709439@jira-lw-us.apache.org>
Subject: [jira] [Updated] (BEAM-1283) DoFn finishBundle should be required
 to specify the window for output
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 19 Apr 2017 22:46:11 -0000


     [ https://issues.apache.org/jira/browse/BEAM-1283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kenneth Knowles updated BEAM-1283:
----------------------------------
    Summary: DoFn finishBundle should be required to specify the window for output  (was: DoFn finishBundle should be required to specify the window)

> DoFn finishBundle should be required to specify the window for output
> ---------------------------------------------------------------------
>
>                 Key: BEAM-1283
>                 URL: https://issues.apache.org/jira/browse/BEAM-1283
>             Project: Beam
>          Issue Type: Bug
>          Components: beam-model, sdk-java-core, sdk-py
>            Reporter: Kenneth Knowles
>              Labels: backward-incompatible
>             Fix For: First stable release
>
>
> The spec is here in Javadoc: https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L128
> "If invoked from {{@StartBundle}} or {{@FinishBundle}}, this will attempt to use the {{WindowFn}} of the input {{PCollection}} to determine what windows the element should be in, throwing an exception if the {{WindowFn}} attempts to access any information about the input element. The output element will have a timestamp of negative infinity."
> This is a collection of caveats that make this method not always technically wrong, but quite a mess. Ideas that reasonable folks have suggested lately:
>  - The {{WindowFn}} cannot actually be applied because {{WindowFn}} is allowed to see the element type. The spec just avoids this by limiting which {{WindowFn}} can be used.
>  - There is no natural output timestamp, so it should always be provided. The spec avoids this by specifying an arbitrary and fairly useless timestamp.
>  - If it is a merging {{WindowFn}} like sessions that has already been merged then you'll just have a bogus proto window regardless of explicit timestamp or not.
> The use cases for these methods are best addressed by state plus window expiry callback, so we should revisit this spec and probably just wipe it.
> There are some rare case where you might need to output from {{FinishBundle}} in a way that is not _actually_ sensitive to bundling (perhaps modulo some downstream notion of equivalence) in which case you had better know what window you are outputting to. Often it should be the global window.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)