kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Connie Chen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-5605) Multiple KStreams being created on application startup are not processing
Date Mon, 17 Jul 2017 22:56:00 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Connie Chen updated KAFKA-5605:
-------------------------------
    Description: 
This is an issue filed in response to this slack discussion
https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627

I have a http service on top of Akka that starts and stops KStreams whose source/sink topics
are configurable. Every time a new KStream is created I post a record to another Kafka topic
`StreamConfig2`. On my application startup, I read from that config topic into a KTable and
create existing KStreams. However, it appears that this startup creation is not deterministic
(ie. sometimes the KStreams get created properly, sometimes not). 

Matthias helpfully suggested that I change `application.id` every time on my application startup
as a workaround. This seems to work for my application (service startup always creates the
existing KStream in config topic), however I can't figure out why it doesn't work when I reuse
`application.id`

I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried increasing the
config there, to no avail. 

Here I've attached logs from my application, expected logs, and kafka server logs. 
https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e

  was:
This is an issue filed in response to this slack discussion
https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627

I have a http service on top of Akka that starts and stops KStreams whose source/sink topics
are configurable. Every time a new KStream is created I post a record to another Kafka topic
`StreamConfig2`. On my application startup, I read from that config topic into a KTable and
create existing KStreams. However, it appears that this startup creation is not deterministic
(ie. sometimes the KStreams get created properly, sometimes not). 

Matthias helpfully suggested that I change `application.id` every time on my application startup
as a workaround. This seems to work for my application (service startup always creates the
existing KStreams), however I can't figure out why it doesn't work when I reuse `application.id`

I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried increasing the
config there, to no avail. 

Here I've attached logs from my application, expected logs, and kafka server logs. 
https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e


> Multiple KStreams being created on application startup are not processing
> -------------------------------------------------------------------------
>
>                 Key: KAFKA-5605
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5605
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Connie Chen
>         Attachments: app-logs.txt, server-logs.txt
>
>
> This is an issue filed in response to this slack discussion
> https://confluentcommunity.slack.com/archives/C48AHTCUQ/p1500079616414627
> I have a http service on top of Akka that starts and stops KStreams whose source/sink
topics are configurable. Every time a new KStream is created I post a record to another Kafka
topic `StreamConfig2`. On my application startup, I read from that config topic into a KTable
and create existing KStreams. However, it appears that this startup creation is not deterministic
(ie. sometimes the KStreams get created properly, sometimes not). 
> Matthias helpfully suggested that I change `application.id` every time on my application
startup as a workaround. This seems to work for my application (service startup always creates
the existing KStream in config topic), however I can't figure out why it doesn't work when
I reuse `application.id`
> I have looked at https://issues.apache.org/jira/browse/KAFKA-5562 and tried increasing
the config there, to no avail. 
> Here I've attached logs from my application, expected logs, and kafka server logs. 
> https://gist.github.com/conniec/baf0c011be8f29a4e09af0ceb136e33e



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message