flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gábor Gévay <gga...@gmail.com>
Subject Re: Cyclic ConnectedStream
Date Sat, 28 Jan 2017 18:17:34 GMT
Hello,

Cyclic dataflows can be built using iterations:
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/datastream_api.html#iterations

Best,
Gábor




2017-01-28 18:39 GMT+01:00 Matt <dromitlabs@gmail.com>:
> I have a ConnectedStream (A) that depends on another ConnectedStream (B),
> which depends on the first one (A).
>
> Simplified code:
>
> predictionStream = input
>   .connect(statsStream)
>   .keyBy(...)
>   .flatMap(CoFlatMapFunction {
>      flatMap1(obj, output) {
>          p = prediction(obj)
>          output.collect(p)
>      }
>      flatMap2(stat, output) {
>          updateModel(stat)
>      }
>   })
>
> statsStream = input2
>   .connect(predictionStream)
>   .keyBy(...)
>   .flatMap(CoFlatMapFunction {
>      flatMap1(obj2, output) {
>         s = getStats(obj2, p)
>         output.collect(s)
>      }
>      flatMap2(prediction, output) {
>         p = prediction
>      }
>   })
>
> I'm guessing this should be possible to achieve, one way would be to add a
> sink on statsStream to save the elements into Kafka and read from that topic
> on predictionStream instead of initializing it with a reference of
> statsStream. But I would rather avoid writing unnecessarily into kafka.
>
> Is there any other way to achieve this?
>
> Thanks,
> Matt

Mime
View raw message