beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kenneth Knowles (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1282) DoFnTester should allow output() calls in start/finishBundle
Date Thu, 19 Jan 2017 21:58:26 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830682#comment-15830682
] 

Kenneth Knowles commented on BEAM-1282:
---------------------------------------

Historical context; this is not really a regression per se:

 - Jan 15, 2015 [20d27ee|https://github.com/apache/beam/commit/20d27ee687576cdb046181f2545b7f59cdae2730]
set up the {{DoFnTester}} to assign windows according to the {{GlobalWindows}} (presumably
because there was no good answer to the question of what {{WindowFn}} to use, and no tests
of this functionality)
 - [33e72de|https://github.com/apache/beam/commit/33e72de6f4eafaa3183585ecf804ccc0267d23b2]
and [f9d8983|https://github.com/apache/beam/commit/f9d89832badccfbb488fddb7fc80c36ce4baae72]
were inconsequential refactors
 - May 31, 2016 [7f2b366|https://github.com/apache/beam/commit/7f2b3669af714f732d6433ba6f94fa0d9855bfaa]
started dropping these outputs silently (presumably overlooked because there was no good answer
to the question of what {{WindowFn}} to use, and no tests of this functionality)
 - Dec 2, 2016 [78ac009|https://github.com/apache/beam/commit/78ac009be743a2e053580e9966f84]
added the crash to call out that this doesn't work correctly and never has

The behavior of always using the global window was wrong (though useful in a ["worse is better"|https://www.jwz.org/doc/worse-is-better.html]
sense), dropping the elements is wrong, and throwing an unsupported operation exception is
not wrong but not useful :-)

> DoFnTester should allow output() calls in start/finishBundle
> ------------------------------------------------------------
>
>                 Key: BEAM-1282
>                 URL: https://issues.apache.org/jira/browse/BEAM-1282
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Daniel Mills
>            Priority: Minor
>              Labels: starter
>
> In a DoFn, users can call output() or outputWithTimestamp() during start/finishBundle.
 This will attempt to deduce a window for the output element based on the current WindowFn
and any timestamp provided. However, DoFnTester always throws an exception if these methods
are called from start/finishBundle (because it does not have a WindowFn, so that will have
to be enhanced)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message