flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: About Interplay of Merged Streams, Output Selectors and Checkpoint Barriers (and Watermarks)
Date Tue, 12 May 2015 12:47:19 GMT
Hi,

Checkpoint barriers are handled directly on top of the network layer and
you are right they work similarly, by blocking input channels until it gets
the barrier from all of them.

A way of implementing this on the operator level would be by adding a way
to ask the inputreader the channel index of the last record. This way the
operator could keep track of the channels from which it has received
records and execute the watermark logic. The IndexedReaders have
implemented the necessarry funcionality but were patched away accidentally
buy some earlier changes (as they were not used anyway)

Adding a union operator is probably an overkill and would pose the same
difficulties when implementing it.

Cheers,
Gyula

On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> Hi Folks,
> as I said in the subject. How will this work? I'm in the process about
> thinking how to implement low watermarks in Streaming. I'm thinking
> that the implementation should be quite similar to how the
> checkpointing barriers will be implemented since they also flush out
> stuff.
>
> Now I'm wondering how this will work with merged Streams and the
> output selectors (split streams). It seems to me that there are a lot
> of paths that elements can take to arrive at operators. The problem I
> have is that an operator can only emit a low watermark itself if it
> knows that all input operators have sent him a low watermark with that
> value (the low watermark is the minimum of the low watermarks of all
> upstream operators). I imagine that the checkpoint barriers exhibit
> the same behaviour.
>
> Do we maybe have to add an explicit union (merge) operator and change
> how split streams are implemented?
>
> What are your thoughts?
>
> Cheers,
> Aljoscha
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message