kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5009) Globally Unique internal topic names when using Joins
Date Tue, 04 Apr 2017 18:16:41 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15955544#comment-15955544

Matthias J. Sax commented on KAFKA-5009:

All topics generated by Kafka Streams are prefixed with {{application.id}} -- thus, multiple
Kafka Streams instances should not conflict on topic names. Can you explain in more detail
what conflicts you did observe? I am also not sure, who this would related to Confluent Schema

> Globally Unique internal topic names when using Joins 
> ------------------------------------------------------
>                 Key: KAFKA-5009
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5009
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Mark Tranter
> We are using multiple different Kafka Streams applications on the back of a single kafka
cluster. This allows each micro-service in our enterprise to consume & process kafka data
to suit its own needs.
> Currently when joining streams, an internal topic is created & named using KStreamsBuilder.newName(prefix);
> ```
>             String thisWindowStreamName = topology.newName(WINDOWED_NAME);
>             String otherWindowStreamName = topology.newName(WINDOWED_NAME);
>             String joinThisName = rightOuter ? topology.newName(OUTERTHIS_NAME) : topology.newName(JOINTHIS_NAME);
>             String joinOtherName = leftOuter ? topology.newName(OUTEROTHER_NAME) : topology.newName(JOINOTHER_NAME);
>             String joinMergeName = topology.newName(MERGE_NAME);
> ```
> This prefix is a constant, and internally an incrementor is used to generate a  unique
(per KStreamBuilder instance) topic name for the topic.
> In situations where multiple KStreamBuilders are in use (for example, multiple different
Kafka Streams applications) we are seeing collisions in topic names.
> Perhaps the join() methods should take a topic prefix overload to allow developers to
provide unique names for these topics. Similar to the way other stateful processors work when
having to provide a store name.

This message was sent by Atlassian JIRA

View raw message