flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hironori Ogibayashi <ogibaya...@gmail.com>
Subject Re: FlinkKafkaConsumer and Kafka topic/partition change
Date Tue, 27 Sep 2016 12:02:40 GMT

Thank you for your quick response!
I am looking forward to that feature. I will periodically check that JIRA.

I am also interested in the Robert's implementation because my use
current case is
system monitoring and scalability has higher priority than correctness.


2016-09-27 19:53 GMT+09:00 Tzu-Li (Gordon) Tai <tzulitai@apache.org>:
> Hi!
> This is definitely a planned feature for the Kafka connectors, there’s a
> JIRA exactly for this [1].
> We’re currently going through some blocking tasks to make this happen, I
> also hope to speed up things over there :)
> Your observation is correct that the Kaka consumer uses “assign()” instead
> of “subscribe()”.
> This is due to the fact that the partition-to-subtask assignment needs to be
> determinate in Flink
> for exactly-once semantics.
> If you’re not concerned about exactly-once and want to experiment around for
> now before [1] comes around,
> I believe Robert has recently implemented a Kafka consumer that uses
> “subscribe()”, so the Kafka
> topics can scale (looping in Robert to provide more info about this one).
> Best Regards,
> Gordon
> [1] https://issues.apache.org/jira/browse/FLINK-4022
> On September 27, 2016 at 6:17:06 PM, Hironori Ogibayashi
> (ogibayashi@gmail.com) wrote:
> Hello,
> I want FlinkKafkaConsumer to follow changes in Kafka topic/partition change.
> This means:
> - When we add partitions to a topic, we want FlinkKafkaConsumer to
> start reading added partitions.
> - We want to specify topics by pattern (e.g accesslog.*), and want
> FlinkKafkaConsumer to start reading new topics if they appeared after
> starting job.
> As long as reading source code and my experiment, FlinkKafkaConsumer
> uses KafkaConsumer.assign() instead of subscribe(), so partitions are
> assigned to each KafkaConsumer instance just once at job starting
> time.
> Is there any way to let FlinkKafkaConsumer follow topic/partition change?
> Regards,
> Hironori Ogibayashi

View raw message