flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AJ Heller ...@drfloob.com>
Subject Merging N parallel/partitioned WindowedStreams together, one-to-one, into a global window stream
Date Thu, 06 Oct 2016 20:39:04 GMT
The goal is:
 * to split data, random-uniformly, across N nodes,
 * window the data identically on each node,
 * transform the windows locally on each node, and
 * merge the N parallel windows into a global window stream, such that one
window from each parallel process is merged into a "global window" aggregate

I've achieved all but the last bullet point, merging one window from each
partition into a globally-aggregated window output stream.

To be clear, a rolling reduce won't work because it would aggregate over
all previous windows in all partitioned streams, and I only need to
aggregate over one window from each partition at a time.

Similarly for a fold.

The closest I have found is ParallelMerge for ConnectedStreams, but I have
not found a way to apply it to this problem. Can flink achieve this? If so,
I'd greatly appreciate a point in the right direction.


View raw message