kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Connie Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5605) Multiple KStreams being created on application startup are not processing
Date Mon, 17 Jul 2017 23:50:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090812#comment-16090812

Connie Chen commented on KAFKA-5605:

[~mjsax] helped me fix the issue. It appears I was sharing `application.id` across all my
`KafkaStream` instances in my application. Each instance of `KafkaStream` under the same `application.id`
must be processing the same topics. So the fix is to create a unique `application.id` per
topic. Then, my service application will have N application.id processing N different topics,
each with a different transformation, and each with its own `KStreamBuilder` and `KafkaStream`


> Multiple KStreams being created on application startup are not processing
> -------------------------------------------------------------------------
>                 Key: KAFKA-5605
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5605
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Connie Chen
>         Attachments: app-logs.txt, server-logs.txt
> This is an issue filed in response to this slack discussion
> https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627
> I have a http service on top of Akka that starts and stops KStreams whose source/sink
topics are configurable. Every time a new KStream is created I post a record to another Kafka
topic `StreamConfig2`. On my application startup, I read from that config topic into a KTable
and iterate records, create one KStream per record. However, it appears that this startup
creation is not deterministic (ie. sometimes the KStreams get created properly, sometimes
not). Also, not all KStreams are created. 
> Matthias helpfully suggested that I change `application.id` every time on my application
startup as a workaround. This seems to work for my application (service startup always creates
the existing KStream in config topic), however I can't figure out why it doesn't work when
I reuse `application.id`
> I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried increasing
the config there, to no avail. 
> Here I've attached logs from my application, expected logs, and kafka server logs. 
> https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e

This message was sent by Atlassian JIRA

View raw message