flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: About Interplay of Merged Streams, Output Selectors and Checkpoint Barriers (and Watermarks)
Date Tue, 12 May 2015 13:02:03 GMT
Watermarks also don't need to flush buffers, they can actually simply queue
in as special stream records, if we want to.

On Tue, May 12, 2015 at 2:53 PM, Gyula Fóra <gyula.fora@gmail.com> wrote:

> Its actually a very different mechanism as watermarks will not block the
> computations
>
> On Tue, May 12, 2015 at 2:48 PM, Matthias J. Sax <
> mjsax@informatik.hu-berlin.de> wrote:
>
> > Hi,
> >
> > I don't understand why we need the same machnism twice in the code...
> > Could checkpoing barrieres and low watermarks be unified (or one build
> > on-top/by-using the other)
> >
> > -Matthias
> >
> >
> > On 05/12/2015 02:47 PM, Gyula Fóra wrote:
> > > Hi,
> > >
> > > Checkpoint barriers are handled directly on top of the network layer
> and
> > > you are right they work similarly, by blocking input channels until it
> > gets
> > > the barrier from all of them.
> > >
> > > A way of implementing this on the operator level would be by adding a
> way
> > > to ask the inputreader the channel index of the last record. This way
> the
> > > operator could keep track of the channels from which it has received
> > > records and execute the watermark logic. The IndexedReaders have
> > > implemented the necessarry funcionality but were patched away
> > accidentally
> > > buy some earlier changes (as they were not used anyway)
> > >
> > > Adding a union operator is probably an overkill and would pose the same
> > > difficulties when implementing it.
> > >
> > > Cheers,
> > > Gyula
> > >
> > > On Tue, May 12, 2015 at 2:40 PM, Aljoscha Krettek <aljoscha@apache.org
> >
> > > wrote:
> > >
> > >> Hi Folks,
> > >> as I said in the subject. How will this work? I'm in the process about
> > >> thinking how to implement low watermarks in Streaming. I'm thinking
> > >> that the implementation should be quite similar to how the
> > >> checkpointing barriers will be implemented since they also flush out
> > >> stuff.
> > >>
> > >> Now I'm wondering how this will work with merged Streams and the
> > >> output selectors (split streams). It seems to me that there are a lot
> > >> of paths that elements can take to arrive at operators. The problem I
> > >> have is that an operator can only emit a low watermark itself if it
> > >> knows that all input operators have sent him a low watermark with that
> > >> value (the low watermark is the minimum of the low watermarks of all
> > >> upstream operators). I imagine that the checkpoint barriers exhibit
> > >> the same behaviour.
> > >>
> > >> Do we maybe have to add an explicit union (merge) operator and change
> > >> how split streams are implemented?
> > >>
> > >> What are your thoughts?
> > >>
> > >> Cheers,
> > >> Aljoscha
> > >>
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message