kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pablo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.
Date Mon, 31 Jul 2017 09:37:01 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107042#comment-16107042
] 

Pablo commented on KAFKA-2729:
------------------------------

Happened to us with timeout workaround:

{code:java}
zookeeper.connection.timeout.ms=10000
zookeeper.session.timeout.ms=10000
{code}

On AWS eu-west-1 on saturday using a 0.10.2.0 cluster of 3 brokers and 3 zks with message
format 0.8.2.0.



> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-2729
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2729
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.2.1
>            Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, we started
seeing a large number of undereplicated partitions. The zookeeper cluster recovered, however
we continued to see a large number of undereplicated partitions. Two brokers in the kafka
cluster were showing this in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker
5: Shrinking ISR for partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker
5: Cached zkVersion [66] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered after a restart.
Our own investigation yielded nothing, I was hoping you could shed some light on this issue.
Possibly if it's related to: https://issues.apache.org/jira/browse/KAFKA-1382 , however we're
using 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message