flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Change sink topology
Date Thu, 06 Jun 2019 12:06:46 GMT
Hi Sergey,

I would not consider this to be a topology change (the sink operator would
still be a Kafka producer).
It seems that dynamic topic selection is possible with a
KeyedSerializationSchema (Check "Advanced Serialization Schema: [1]).

Best,
Fabian

[1]
https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/connectors/kafka.html#kafka-producer

Am Do., 6. Juni 2019 um 13:50 Uhr schrieb Smirnov Sergey Vladimirovich <
s.smirnov1@tinkoff.ru>:

> Hi flink,
>
>
>
> Im wonder, is it possible to dynamically (while job running) change sink
> topology* - by adding new sink on the fly?
>
> Say, we have input stream and by analyzing message property we decided to
> put this message into some kafka topic, i.e. choosen_topic =
> function(message.property).
>
> Simplifying: sink_topic = ‘logger_group_’+message.groupId
>
> When job was launched we don`t know list of all possible groupId.
>
> These sink topics (and groupId) are created (and even removed) dynamically
> upon executing some external rules during the entire job period.
>
>
>
>
>
> Best, Sergey
>

Mime
View raw message