flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <mj...@informatik.hu-berlin.de>
Subject Re: About Interplay of Merged Streams, Output Selectors and Checkpoint Barriers (and Watermarks)
Date Tue, 12 May 2015 12:48:35 GMT

I don't understand why we need the same machnism twice in the code...
Could checkpoing barrieres and low watermarks be unified (or one build
on-top/by-using the other)


On 05/12/2015 02:47 PM, Gyula Fóra wrote:
> Hi,
> Checkpoint barriers are handled directly on top of the network layer and
> you are right they work similarly, by blocking input channels until it gets
> the barrier from all of them.
> A way of implementing this on the operator level would be by adding a way
> to ask the inputreader the channel index of the last record. This way the
> operator could keep track of the channels from which it has received
> records and execute the watermark logic. The IndexedReaders have
> implemented the necessarry funcionality but were patched away accidentally
> buy some earlier changes (as they were not used anyway)
> Adding a union operator is probably an overkill and would pose the same
> difficulties when implementing it.
> Cheers,
> Gyula
> On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljoscha@apache.org>
> wrote:
>> Hi Folks,
>> as I said in the subject. How will this work? I'm in the process about
>> thinking how to implement low watermarks in Streaming. I'm thinking
>> that the implementation should be quite similar to how the
>> checkpointing barriers will be implemented since they also flush out
>> stuff.
>> Now I'm wondering how this will work with merged Streams and the
>> output selectors (split streams). It seems to me that there are a lot
>> of paths that elements can take to arrive at operators. The problem I
>> have is that an operator can only emit a low watermark itself if it
>> knows that all input operators have sent him a low watermark with that
>> value (the low watermark is the minimum of the low watermarks of all
>> upstream operators). I imagine that the checkpoint barriers exhibit
>> the same behaviour.
>> Do we maybe have to add an explicit union (merge) operator and change
>> how split streams are implemented?
>> What are your thoughts?
>> Cheers,
>> Aljoscha

View raw message