kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt Farmer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-6214) Using standby replicas with an in memory state store causes Streams to crash
Date Thu, 16 Nov 2017 15:40:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255490#comment-16255490

Matt Farmer commented on KAFKA-6214:

Is the answer here, then, to just remove that {{if}} statement?

> Using standby replicas with an in memory state store causes Streams to crash
> ----------------------------------------------------------------------------
>                 Key: KAFKA-6214
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6214
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions:
>            Reporter: Matt Farmer
>              Labels: statestore
> We decided to start experimenting with Standby Replicas of our State Stores by setting
the following configuration setting:
> {code}
> num.standby.replicas=1
> {code}
> Most applications did okay with this except for one that used an in memory state store
instead of a persistent state store. With the new configuration, the first instance of this
application booted fine. When the second instance came up, both instances crashed with the
following exception:
> {code}
> java.lang.IllegalStateException: Consumer is not subscribed to any topics or assigned
any partitions
>         at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1037)
>         at org.apache.kafka.streams.processor.internals.StreamThread.maybeUpdateStandbyTasks(StreamThread.java:752)
>         at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:524)
>         at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:480)
>         at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:457)
> {code}
> Monit attempted to restart both instances but they would just continue to crash over
and over again. The state store in our problematic application is declared like so:
> {code}
> Stores
>     .create("TheStateStore")
>     .withStringKeys()
>     .withStringValues()
>     .inMemory()
>     .build()
> {code}
> Luckily we had a config switch in place that could turn on an alternate, persistent state
store. As soon as we flipped to the persistent state store, things started working as we expected.

This message was sent by Atlassian JIRA

View raw message