flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From burgesschen <tchen...@bloomberg.net>
Subject Multiple operators to the same sink
Date Thu, 08 Nov 2018 00:20:52 GMT
Hi Guys! I'm designing a topology where multiple operators should forward the
messages to the same sink.

For example I have Operator A,B,C,D,E. I want A,B,C to forward to Sink1 and
D, E to forward to Sink2.

My options are

1. Union A, B and C. then add Sink1 to them. Similarly for D and E. However,
the current framework out team has builds each operator individually. There
is nothing outside of the operators 
that has the knowledge of their destination sink. It means we need to build
something on the job level to union the operators.

2. have each operator output to a side output tag. A,B, and C will output to
tag "sink1", And have a singleton sink1 to consume from tag "sink1".
Similarly for sink2. My concern here is that 'it feels hacky', since those
messages are not really side outputs.

is this a legitimate use case for output tag or not? Is there a better way
to achieve this? Thank you!

Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

View raw message