flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hironori Ogibayashi <ogibaya...@gmail.com>
Subject FlinkKafkaConsumer and Kafka topic/partition change
Date Tue, 27 Sep 2016 10:17:02 GMT

I want FlinkKafkaConsumer to follow changes in Kafka topic/partition change.
This means:
- When we add partitions to a topic, we want FlinkKafkaConsumer to
start reading added partitions.
- We want to specify topics by pattern (e.g accesslog.*), and want
FlinkKafkaConsumer to start reading new topics if they appeared after
starting job.

As long as reading source code and my experiment, FlinkKafkaConsumer
uses KafkaConsumer.assign() instead of subscribe(), so partitions are
assigned to each KafkaConsumer instance just once at job starting

Is there any way to let FlinkKafkaConsumer follow topic/partition change?

Hironori Ogibayashi

View raw message