kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Blake Miller (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4936) Allow dynamic routing of output records
Date Fri, 02 Mar 2018 01:28:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382969#comment-16382969

Blake Miller commented on KAFKA-4936:

That is a very good point about the potential to create oodles of topics unintentionally.
I don't think you're overthinking it... there is significantly more complexity to it than
I recognized at first, and I was not considering automatic topic creation. I pondered it a
bit more just now, and came to the tentative conclusion that this feature could be implemented
in a safe and simple way *only* in the absence of automatic topic creation.

I personally do not understand what value automatic topic creation offers, it seems to
me to abstract over very important details that the Kafka user should think about (partitioning,
at least), and also to lower the bar for mistakes or bad code doing damage, in order to
gain what seems like a very tiny bit of convenience. Perhaps somebody who understands the value
proposition of automatic topic creation can comment on its relevance to this issue.

Regarding the implementation details, I hadn't thought very deeply into it myself, but what
you say makes sense, and even without automatic topic creation, the implementation would have
to handle the eventuality of the topic not existing.

I wonder what happens in the current implementation if I give a nonexistent output topic to
a StreamBuilder and then .start() it, specifically would it be a runtime error somewhere in
a SinkNode after the Kafka Streams app is started, or would StreamBuilder.build() complain?
I will give that a try.



> Allow dynamic routing of output records
> ---------------------------------------
>                 Key: KAFKA-4936
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4936
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Priority: Major
>              Labels: needs-kip
> Currently, all used output topics must be know beforehand, and thus, it's not possible
to send output records to topic in a dynamic fashion.
> There have been couple of request for this feature and we should consider adding it.
There are many open questions however, with regard to topic creation and configuration (replication
factor, number of partitions) etc.

This message was sent by Atlassian JIRA

View raw message