kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (Jira)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-10892) Add Topology#connectSourceStoreAndTopic as a public method
Date Tue, 29 Dec 2020 20:59:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-10892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Matthias J. Sax updated KAFKA-10892:
    Issue Type: New Feature  (was: Improvement)

> Add Topology#connectSourceStoreAndTopic as a public method
> ----------------------------------------------------------
>                 Key: KAFKA-10892
>                 URL: https://issues.apache.org/jira/browse/KAFKA-10892
>             Project: Kafka
>          Issue Type: New Feature
>          Components: streams
>    Affects Versions: 2.7.0
>            Reporter: Tomohiro Hashidate
>            Priority: Major
>              Labels: need-kip
> I want Topology#connectSourceStoreAndTopic.
> Because I want to use a topic as a source topic directly without a redundant changelog
topic for not only KeyValueStore but also WindowStore.
> This issue is similar to [KAFKA-6840], but is a suggestion for a more general approach
> {code:java}
>     public synchronized Topology connectSourceStoreAndTopic(final String sourceStoreName,
>                                                             final String topic) {
>         internalTopologyBuilder.connectSourceStoreAndTopic(sourceStoreName, topic);
>         return this;
>     }
> {code}
> h3. Background
> I want to use a topic as a source topic for WindowStore because using WindowStore is
suitable for the feature that I'm implementing.
>  The records stored in the topic are aggregated with a time window by another application.
The size of the topic is over 10TB.
>  I want to use the topic as a source topic for WindowStore directly.
>  But, I cannot do so on the current interface. 
>  I need a redundant topic only for storing the records into WindowStore.
> If this API is public, I can use topics incoming from other applications (not only Kafka
Streams applications) as source topics for any StateStore implementation without redundant
changelog topics.
> Of course, I need to implement a processor for storing incoming records in such a case.
>  But I think it's not difficult.
> Please consider this.

This message was sent by Atlassian Jira

View raw message