flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Thoughts About Streaming
Date Mon, 22 Jun 2015 19:34:17 GMT
Hi,
with people proposing changes to the streaming part I also wanted to throw
my hat into the ring. :D

During the last few months, while I was getting acquainted with the
streaming system, I wrote down some thoughts I had about how things could
be improved. Hopefully, they are in somewhat coherent shape now, so please
have a look if you are interested in this:
https://docs.google.com/document/d/1rSoHyhUhm2IE30o5tkR8GEetjFvMRMNxvsCfoPsW6_4/edit?usp=sharing

This mostly covers:
 - Timestamps assigned at sources
 - Out-of-order processing of elements in window operators
 - API design

Please let me know what you think. Comment in the document or here in the
mailing list.

I have a PR in the makings that would introduce source timestamps and
watermarks for keeping track of them. I also hacked a proof-of-concept of a
windowing system that is able to process out-of-order elements using a
FlatMap operator. (It uses panes to perform efficient pre-aggregations.)

Cheers,
Aljoscha

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message