flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: About Interplay of Merged Streams, Output Selectors and Checkpoint Barriers (and Watermarks)
Date Tue, 12 May 2015 12:53:38 GMT
Its actually a very different mechanism as watermarks will not block the
computations

On Tue, May 12, 2015 at 2:48 PM, Matthias J. Sax <
mjsax@informatik.hu-berlin.de> wrote:

> Hi,
>
> I don't understand why we need the same machnism twice in the code...
> Could checkpoing barrieres and low watermarks be unified (or one build
> on-top/by-using the other)
>
> -Matthias
>
>
> On 05/12/2015 02:47 PM, Gyula Fóra wrote:
> > Hi,
> >
> > Checkpoint barriers are handled directly on top of the network layer and
> > you are right they work similarly, by blocking input channels until it
> gets
> > the barrier from all of them.
> >
> > A way of implementing this on the operator level would be by adding a way
> > to ask the inputreader the channel index of the last record. This way the
> > operator could keep track of the channels from which it has received
> > records and execute the watermark logic. The IndexedReaders have
> > implemented the necessarry funcionality but were patched away
> accidentally
> > buy some earlier changes (as they were not used anyway)
> >
> > Adding a union operator is probably an overkill and would pose the same
> > difficulties when implementing it.
> >
> > Cheers,
> > Gyula
> >
> > On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljoscha@apache.org>
> > wrote:
> >
> >> Hi Folks,
> >> as I said in the subject. How will this work? I'm in the process about
> >> thinking how to implement low watermarks in Streaming. I'm thinking
> >> that the implementation should be quite similar to how the
> >> checkpointing barriers will be implemented since they also flush out
> >> stuff.
> >>
> >> Now I'm wondering how this will work with merged Streams and the
> >> output selectors (split streams). It seems to me that there are a lot
> >> of paths that elements can take to arrive at operators. The problem I
> >> have is that an operator can only emit a low watermark itself if it
> >> knows that all input operators have sent him a low watermark with that
> >> value (the low watermark is the minimum of the low watermarks of all
> >> upstream operators). I imagine that the checkpoint barriers exhibit
> >> the same behaviour.
> >>
> >> Do we maybe have to add an explicit union (merge) operator and change
> >> how split streams are implemented?
> >>
> >> What are your thoughts?
> >>
> >> Cheers,
> >> Aljoscha
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message