kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4936) Allow dynamic routing of output records
Date Fri, 02 Mar 2018 02:36:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383038#comment-16383038

Matthias J. Sax commented on KAFKA-4936:

If you provide an unknown topic name for an output topic, the topic with either be created
if auto-topic create is enabled, or a runtime error occurs. StreamsBuilder does not check
the existence of topics.

I tend to agree, that auto.topic create is a "tricky feature". The main question is the following:
if we argue, that auto topic creation should not be used, this implies that all output topic
must be know in advance because users need to create them. And thus, users can also use the
{{branch()}} operator and don't need "auto routing" in the first place. Thus, it seems that
people asking for this feature actually assume that auto-topic create is enabled – otherwise
I cannot explain why one would need this feature?

I am still not convinced about the overall story / need for this. We also would need a KIP
and a discussion on the mailing list. Thus, it might make more sense to do this discussion
first, before starting to write the code – we might decide that we don't want this for the
mentioned reasons. When we go the feature request, we just create the Jira to track it –
but in retrospective, I am not sure about it anymore.

> Allow dynamic routing of output records
> ---------------------------------------
>                 Key: KAFKA-4936
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4936
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Priority: Major
>              Labels: needs-kip
> Currently, all used output topics must be know beforehand, and thus, it's not possible
to send output records to topic in a dynamic fashion.
> There have been couple of request for this feature and we should consider adding it.
There are many open questions however, with regard to topic creation and configuration (replication
factor, number of partitions) etc.

This message was sent by Atlassian JIRA

View raw message