Mailing-List: contact dev-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kafka.apache.org
Date: Wed, 23 Nov 2016 15:32:00 +0000 (UTC)
From: "derek (JIRA)" <jira@apache.org>
To: dev@kafka.apache.org
Message-ID: <JIRA.12910010.1446562853000.351643.1479915120340@Atlassian.JIRA>
In-Reply-To: <JIRA.12910010.1446562853000@Atlassian.JIRA>
References: <JIRA.12910010.1446562853000@Atlassian.JIRA> <JIRA.12910010.1446562853424@arcas>
Subject: [jira] [Issue Comment Deleted] (KAFKA-2729) Cached zkVersion not
 equal to that in zookeeper, broker not recovering.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 23 Nov 2016 15:32:05 -0000


     [ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

derek updated KAFKA-2729:
-------------------------
    Comment: was deleted

(was: I'm on 0.10.1.0 and seeing the same thing. Maybe related to [~cmolter] is saying above, what we see in the logs just prior to a broker becoming under-replicated is a flurry of

{noformat}
org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition.
{noformat}

messages. After that we see a bunch of activity around adding and removing fetchers, then it goes into the infinite ISR shrink loop. The only way we can recover is to restart.)

> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-2729
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2729
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.2.1
>            Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, we started seeing a large number of undereplicated partitions. The zookeeper cluster recovered, however we continued to see a large number of undereplicated partitions. Two brokers in the kafka cluster were showing this in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker 5: Shrinking ISR for partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker 5: Cached zkVersion [66] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered after a restart. Our own investigation yielded nothing, I was hoping you could shed some light on this issue. Possibly if it's related to: https://issues.apache.org/jira/browse/KAFKA-1382 , however we're using 0.8.2.1.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)