flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: About Interplay of Merged Streams, Output Selectors and Checkpoint Barriers (and Watermarks)
Date Tue, 12 May 2015 12:50:25 GMT
I would like to refrain from adding additional tasks as much as possible.

I agree with Gyula that extending the reader to track watermarks and call a
handler whenever the watermark advances would be a nice way to implement
this.

On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> Hi Folks,
> as I said in the subject. How will this work? I'm in the process about
> thinking how to implement low watermarks in Streaming. I'm thinking
> that the implementation should be quite similar to how the
> checkpointing barriers will be implemented since they also flush out
> stuff.
>
> Now I'm wondering how this will work with merged Streams and the
> output selectors (split streams). It seems to me that there are a lot
> of paths that elements can take to arrive at operators. The problem I
> have is that an operator can only emit a low watermark itself if it
> knows that all input operators have sent him a low watermark with that
> value (the low watermark is the minimum of the low watermarks of all
> upstream operators). I imagine that the checkpoint barriers exhibit
> the same behaviour.
>
> Do we maybe have to add an explicit union (merge) operator and change
> how split streams are implemented?
>
> What are your thoughts?
>
> Cheers,
> Aljoscha
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message