gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GEARPUMP-23) Add DSL window (time series) support
Date Thu, 12 May 2016 01:01:12 GMT

    [ https://issues.apache.org/jira/browse/GEARPUMP-23?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281081#comment-15281081
] 

Manu Zhang commented on GEARPUMP-23:
------------------------------------

Thanks, Kam.

I read this paper some while ago. It has a great summary of algebraic properties of aggregate
operations. For example, count is associative, invertible and commutative, and thus fits for
out-of-order processing with accumulating and distracting. Other operations may have more
restrictions and harder to handle in a window aggregation.  It's worthwhile to explore how
to implement the approach of the paper on Gearpump for the long term.

The shortcut to go now is to stick to the Beam model, although my impression (which could
be wrong) is that it requires the aggregation to be associative and commutative.

> Add DSL window (time series) support 
> -------------------------------------
>
>                 Key: GEARPUMP-23
>                 URL: https://issues.apache.org/jira/browse/GEARPUMP-23
>             Project: Apache Gearpump
>          Issue Type: New Feature
>          Components: streaming
>    Affects Versions: 0.8.0
>            Reporter: Kam Kasravi
>            Assignee: Manu Zhang
>             Fix For: 0.8.1
>
>
> Time series support is need for both GEARPUMP-21 and GEARPUMP-22. This was originally
github issue  [1917|https://github.com/gearpump/gearpump/issues/1917]. Other realtime streaming
engines support time windows such as [Flink|https://flink.apache.org/news/2015/12/04/Introducing-windows.html],
[Beam|https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101] and a future release
of [Spark|https://issues.apache.org/jira/secure/attachment/12793410/StructuredStreamingProgrammingAbstractionSemanticsandAPIs-ApacheJIRA.pdf].
A comparison matrix is found [here|https://docs.google.com/spreadsheets/d/1OM077lZBARrtUi6g0X0O0PHaIbFKCD6v0djRefQRE1I/edit#gid=995660187]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message