flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject [DISCUSS] Change Streaming Operators to be Push-Only
Date Tue, 05 May 2015 11:37:37 GMT
Hi Folks,
while working on introducing source-assigned timestamps into streaming
(https://issues.apache.org/jira/browse/FLINK-1967) I thought about how
the punctuations (low watermarks) can be pushed through the system.
The problem is, that operators can have two ways of getting input: 1.
They read directly from input iterators, and 2. They act as a
Collector and get elements via collect() from the previous operator in
a chain.

This makes it hard to push things through a chain that are not
elements, such as barriers and/or punctuations.

I propose to change all streaming operators to be push based, with a
slightly improved interface: In addition to collect(), which I would
call receiveElement() I would add receivePunctuation() and
receiveBarrier(). The first operator in the chain would also get data
from the outside invokable that reads from the input iterator and
calls receiveElement() for the first operator in a chain.

What do you think? I would of course be willing to implement this myself.


View raw message