flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gyula Fora (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-1618) Add parallel time discretisation for time-window transformations
Date Mon, 02 Mar 2015 09:01:04 GMT
Gyula Fora created FLINK-1618:
---------------------------------

             Summary: Add parallel time discretisation for time-window transformations 
                 Key: FLINK-1618
                 URL: https://issues.apache.org/jira/browse/FLINK-1618
             Project: Flink
          Issue Type: Improvement
          Components: Streaming
            Reporter: Gyula Fora


Currently discretizers for all windowing policies including time are executed with parallelism
1 when they define global windows. (for instance: sum of the last 10 minutes) 

While this is necessary for arbitrary policies like delta based or user-defined policies.
Some discretizers such as Time can be implemented in a distributed fashion.

Distributed time discretisers (and other types) can be implemented in the following way:

-The discretisers should create StreamWindow s with incrementally increasing ID-s starting
from the same value so that it is possible to merge them after the transformation
- The partitioner for each discretizer should send the number of partitions created to the
merger (the merger should be aware of the number of partitioners present to wait for all the
information)
- Based on all the partitioning info the merger can merge the windows properly afterwards





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message