flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From satya brat <bratsatya...@gmail.com>
Subject Re: Creating Kafka Topic dynamically in Flink
Date Mon, 01 Jun 2020 10:33:26 GMT
Prasanna,
You might want to check the kafka broker configs where
'auto.create.topics.enable' helps with creating a new topic whenever a new
message with non existent topic is published.
https://kafka.apache.org/documentation/#brokerconfigs

I am not too sure about pitfalls if any.

On Mon, Jun 1, 2020 at 3:20 PM Leonard Xu <xbjtdcq@gmail.com> wrote:

> Hi, kumar
>
> Sorry for missed the original question, I think we can not create topic
> dynamically current, creating topic should belong to control flow rather a
> data flow, and user may has some custom configurations of the topic from my
> understanding. Maybe you need implement the logic of check/create/manage
> topic in your custom SinkFunction so that the topic can create dynamically
> in runtime.
>
> Best,
> Leonard Xu
>
> 在 2020年6月1日,17:02,Prasanna kumar <prasannakumarramani@gmail.com> 写道:
>
> Leaonard,
>
> Thanks for the reply and would look into those options.
> But as for the original question, could we create a topic dynamically when
> required .
>
> Prasanna.
>
> On Mon, Jun 1, 2020 at 2:18 PM Leonard Xu <xbjtdcq@gmail.com> wrote:
>
>> Hi, kumar
>>
>> Flink support consume/produce from/to multiple kafka topics[1], in your
>> case you can implement KeyedSerializationSchema(legacy interface) or
>> KafkaSerializationSchema[2] to make one producer instance support send data
>> to multiple topics. There is an ITCase you can reference[3].
>>
>>
>> Best,
>> Leonard Xu
>>
>> [1]
>> https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/kafka.html#kafka-producer
>> [2]
>> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/KafkaSerializationSchema.java
>>
>> [3]
>> https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.11/src/test/java/org/apache/flink/streaming/connectors/kafka/Kafka011ITCase.java#L126
>>
>>
>> 在 2020年6月1日,15:35,Prasanna kumar <prasannakumarramani@gmail.com>
写道:
>>
>> Hi,
>>
>> I have Use Case where i read events from a Single kafka Stream comprising
>> of JSON messages.
>>
>> Requirement is to split the stream into multiple output streams based on
>> some criteria say based on Type of Event or Based on Type and Customer
>> associated with the event.
>>
>> We could achieve the splitting of stream using Side outputs as i have
>> seen in the documentation.
>>
>> Our business environment is such that there could be new event types
>> flowing in and would the Flink Kafka producer create the topics dynamically
>> based on the inflowing events. I did not see any documentation saying
>> that it could create.
>>
>> Or should it be always pre created by running a script separately. (Not a
>> good scalable practice in our case)
>>
>> Thanks,
>> Prasanna.
>>
>>
>>
>

Mime
View raw message