flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyf...@apache.org>
Subject Stream iteration head as ConnectedDataStream
Date Fri, 26 Jun 2015 11:35:20 GMT
Hey!

Now that we are implementing more and more applications for streaming that
use iterations we realized a huge shortcoming of the current iteration api.
Currently it only allows to feedback data of the same type to the iteration
head.

This makes sense because the operators are typed but makes it awkward if we
indeed want to use a different feedback (such as a model syncing for
 machine learning applications). To do this developers need to use wrapper
types and flags to distinguish the inputs.

I propose to add the possibility to tread the original input of the
iteration head operator and the feedback stream as a ConnectedDataStream so
we can apply operators such as CoMap, CoFlatMap etc. This helps
distinguishing the inputs and also allows different feedback types to be
used. I believe this change is inevitable if we want to write elegant
applications without unnecessary wrapper types.

I made a PR <https://github.com/apache/flink/pull/870> already that
introduces this functionality (it is a very small change in fact).

Cheers,
Gyula

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message