flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Thoughts About Streaming
Date Tue, 23 Jun 2015 16:29:47 GMT
I also don't like big changes but sometimes they are necessary. The reason
why I'm so adamant about out-of-order processing is that out-of-order
elements are not some exception that occurs once in a while; they occur
constantly in a distributed system. For example, in this:
https://gist.github.com/aljoscha/a367012646ab98208d27 the resulting windows
are completely bogus because the current windowing system assumes elements
to globally arrive in order, which is simply not true. (The example has a
source that generates increasing integers. Then these pass through a map
and are unioned with the original DataStream before a window operator.)
This simulates elements arriving from different operators at a windowing
operator. The example is also DOP=1, I imagine this to get worse with
higher DOP.

What do you mean by costly? As I said, I have a proof-of-concept windowing
operator that can handle out-or-order elements. This is an example using
the current Flink API: https://gist.github.com/aljoscha/f8dce0691732e344bbe8.
(It is an infinite source of tuples and a 5 second window operator that
counts the tuples.) The first problem is that this code deadlocks because
of the thread that emits fake elements. If I disable the fake element code
it works, but the throughput using my mockup is 4 times higher . The gap
widens dramatically if the window size increases.

So, it actually increases performance (unless I'm making a mistake in my
explorations) and can handle elements that arrive out-of-order (which
happens basically always in a real-world windowing use-cases).


On Tue, 23 Jun 2015 at 12:51 Stephan Ewen <sewen@apache.org> wrote:

> What I like a lot about Aljoscha's proposed design is that we need no
> different code for "system time" vs. "event time". It only differs in where
> the timestamps are assigned.
>
> The OOP approach also gives you the semantics of total ordering without
> imposing merges on the streams.
>
> On Tue, Jun 23, 2015 at 10:43 AM, Matthias J. Sax <
> mjsax@informatik.hu-berlin.de> wrote:
>
> > I agree that there should be multiple alternatives the user(!) can
> > choose from. Partial out-of-order processing works for many/most
> > aggregates. However, if you consider Event-Pattern-Matching, global
> > ordering in necessary (even if the performance penalty might be high).
> >
> > I would also keep "system-time windows" as an alternative to "source
> > assigned ts-windows".
> >
> > It might also be interesting to consider the following paper for
> > overlapping windows: "Resource sharing in continuous sliding-window
> > aggregates"
> >
> > > https://dl.acm.org/citation.cfm?id=1316720
> >
> > -Matthias
> >
> > On 06/23/2015 10:37 AM, Gyula Fóra wrote:
> > > Hey
> > >
> > > I think we should not block PRs unnecessarily if your suggested changes
> > > might touch them at some point.
> > >
> > > Also I still think we should not put everything in the Datastream
> because
> > > it will be a huge mess.
> > >
> > > Also we need to agree on the out of order processing, whether we want
> it
> > > the way you proposed it(which is quite costly). Another alternative
> > > approach there which fits in the current windowing is to filter out if
> > > order events and apply a special handling operator on them. This would
> be
> > > fairly lightweight.
> > >
> > > My point is that we need to consider some alternative solutions. And we
> > > should not block contributions along the way.
> > >
> > > Cheers
> > > Gyula
> > >
> > > On Tue, Jun 23, 2015 at 9:55 AM Aljoscha Krettek <aljoscha@apache.org>
> > > wrote:
> > >
> > >> The reason I posted this now is that we need to think about the API
> and
> > >> windowing before proceeding with the PRs of Gabor (inverse reduce) and
> > >> Gyula (removal of "aggregate" functions on DataStream).
> > >>
> > >> For the windowing, I think that the current model does not work for
> > >> out-of-order processing. Therefore, the whole windowing infrastructure
> > will
> > >> basically have to be redone. Meaning also that any work on the
> > >> pre-aggregators or optimizations that we do now becomes useless.
> > >>
> > >> For the API, I proposed to restructure the interactions between all
> the
> > >> different *DataStream classes and grouping/windowing. (See API section
> > of
> > >> the doc I posted.)
> > >>
> > >> On Mon, 22 Jun 2015 at 21:56 Gyula Fóra <gyula.fora@gmail.com> wrote:
> > >>
> > >>> Hi Aljoscha,
> > >>>
> > >>> Thanks for the nice summary, this is a very good initiative.
> > >>>
> > >>> I added some comments to the respective sections (where I didnt fully
> > >> agree
> > >>> :).).
> > >>> At some point I think it would be good to have a public hangout
> session
> > >> on
> > >>> this, which could make a more dynamic discussion.
> > >>>
> > >>> Cheers,
> > >>> Gyula
> > >>>
> > >>> Aljoscha Krettek <aljoscha@apache.org> ezt írta (időpont: 2015.
jún.
> > >> 22.,
> > >>> H, 21:34):
> > >>>
> > >>>> Hi,
> > >>>> with people proposing changes to the streaming part I also wanted
to
> > >>> throw
> > >>>> my hat into the ring. :D
> > >>>>
> > >>>> During the last few months, while I was getting acquainted with
the
> > >>>> streaming system, I wrote down some thoughts I had about how things
> > >> could
> > >>>> be improved. Hopefully, they are in somewhat coherent shape now,
so
> > >>> please
> > >>>> have a look if you are interested in this:
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> https://docs.google.com/document/d/1rSoHyhUhm2IE30o5tkR8GEetjFvMRMNxvsCfoPsW6_4/edit?usp=sharing
> > >>>>
> > >>>> This mostly covers:
> > >>>>  - Timestamps assigned at sources
> > >>>>  - Out-of-order processing of elements in window operators
> > >>>>  - API design
> > >>>>
> > >>>> Please let me know what you think. Comment in the document or here
> in
> > >> the
> > >>>> mailing list.
> > >>>>
> > >>>> I have a PR in the makings that would introduce source timestamps
> and
> > >>>> watermarks for keeping track of them. I also hacked a
> proof-of-concept
> > >>> of a
> > >>>> windowing system that is able to process out-of-order elements
> using a
> > >>>> FlatMap operator. (It uses panes to perform efficient
> > >> pre-aggregations.)
> > >>>>
> > >>>> Cheers,
> > >>>> Aljoscha
> > >>>>
> > >>>
> > >>
> > >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message