beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-542) Spark batch interval should be a configuration instead of an interpretation of the Pipeline's windows
Date Wed, 10 Aug 2016 17:58:20 GMT

    [ https://issues.apache.org/jira/browse/BEAM-542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415676#comment-15415676
] 

ASF GitHub Bot commented on BEAM-542:
-------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-beam/pull/808


> Spark batch interval should be a configuration instead of an interpretation of the Pipeline's
windows
> -----------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-542
>                 URL: https://issues.apache.org/jira/browse/BEAM-542
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-spark
>            Reporter: Amit Sela
>            Assignee: Amit Sela
>
> Currently, the SparkRunner extracts the batch interval from the duration of the first
window. 
> This is wrong in several ways:
> # GlobalWindow pipelines
> # It's an engine specific property and should not be expressed as a part of the logic
but rather as a configuration for execution of the pipeline.
> # Effectively forces the definition of Fixed/SlidingWindows even when they are not needed
(stateless processing), which also makes the pipeline code not portable.    



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message