kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sophie Blee-Goldman (Jira)" <j...@apache.org>
Subject [jira] [Created] (KAFKA-10395) TopologyTestDriver does not work with dynamic topic routing
Date Wed, 12 Aug 2020 23:58:00 GMT
Sophie Blee-Goldman created KAFKA-10395:

             Summary: TopologyTestDriver does not work with dynamic topic routing
                 Key: KAFKA-10395
                 URL: https://issues.apache.org/jira/browse/KAFKA-10395
             Project: Kafka
          Issue Type: Bug
          Components: streams
            Reporter: Sophie Blee-Goldman

The TopologyTestDriver#read(topic) methods all call #getRecordsQueue which checks 

final Queue<ProducerRecord<byte[], byte[]>> outputRecords = outputRecordsByTopic.get(topicName);
if (outputRecords == null) {
    if (!processorTopology.sinkTopics().contains(topicName)) {
        throw new IllegalArgumentException("Unknown topic: " + topicName); 
The outputRecordsByTopic map keeps track of all topics that are actually produced to, but
obviously doesn't capture any topics that haven't yet received output. The `processorTopology#sinkTopics`
is supposed to account for that by checking to make sure the topic is actually registered
in the topology, and throw an exception if not in case the user supplied the wrong topic name
to read from. 

Unfortunately the TopicNameExtractor allows for dynamic routing of records to any topic, so
the topology isn't aware of all the possible output topics. If trying to read from one of
these topics that happens to not have received any output yet, the test will throw the above
misleading IllegalArgumentException.

We could just relax this check, but warning users who may actually have accidentally passed
in the wrong topic to read from seems quite useful. A better solution would be to require
registering all possible output topics to the TTD up front. This would obviously require a
KIP, but it would be a very small one and shouldn't be too much trouble


This message was sent by Atlassian Jira

View raw message