flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-4789) Avoid Kafka partition discovery on restore and share consumer instance for discovery and data consumption
Date Mon, 10 Oct 2016 13:05:20 GMT
Robert Metzger created FLINK-4789:
-------------------------------------

             Summary: Avoid Kafka partition discovery on restore and share consumer instance
for discovery and data consumption
                 Key: FLINK-4789
                 URL: https://issues.apache.org/jira/browse/FLINK-4789
             Project: Flink
          Issue Type: Improvement
          Components: Kafka Connector
    Affects Versions: 1.2.0
            Reporter: Robert Metzger


As part of FLINK-4379, the Kafka partition discovery was moved from the Constructor to the
open() method. This is in general a good change, as outlined in FLINK-4155, as it allows us
to detect new partitions and topics based on regex on the fly.

However, currently the partitions are discovered on restore as well. 
Also, the {{FlinkKafkaConsumer09.getKafkaPartitions()}} is creating a separate {{KafkaConsumer}}
just for the partition discovery.
Since the partition discovery happens on the task managers now, we can use the regular {{KafkaConsumer}}
instance, which is used for data retrieval as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message