kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Blake Miller (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4936) Allow dynamic routing of output records
Date Fri, 02 Mar 2018 19:16:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384010#comment-16384010
] 

Blake Miller commented on KAFKA-4936:
-------------------------------------

{quote}this implies that all output topic must be know in advance because users need to create
them. And thus, users can also use the branch() operator and don't need "auto routing" in
the first place
{quote}
I think that is not quite correct ^

The set of output topics might be open, (not known at topology-creation time), but still created
explicitly.

They may be created explicitly outside of the Kafka Streams app, while the Kafka Streams app
is running, for example.

That's precisely what drew my interest to this potential feature: I had planned to use Kafka
Streams for a simple service I was building, that has that property ^, and found that this
limitation prevented me from doing so. So I fell back on a plain Producer/Consumer implementation,
which is fine, but I'm pretty convinced that there is utility to this.

I will organize my thoughts on it and start a discussion on the mailing list. Thanks for helping
me get started!

> Allow dynamic routing of output records
> ---------------------------------------
>
>                 Key: KAFKA-4936
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4936
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>            Priority: Major
>              Labels: needs-kip
>
> Currently, all used output topics must be know beforehand, and thus, it's not possible
to send output records to topic in a dynamic fashion.
> There have been couple of request for this feature and we should consider adding it.
There are many open questions however, with regard to topic creation and configuration (replication
factor, number of partitions) etc.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message