kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Francesco vigotti (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-2729) Cached zkVersion not equal to that in zookeeper, broker not recovering.
Date Fri, 13 Oct 2017 16:01:01 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203749#comment-16203749
] 

Francesco vigotti commented on KAFKA-2729:
------------------------------------------

After having lost 2 days on this I've reset whole cluster, stopped all kafka brokers, stopped
zookeeper cluster, delete all directories,stopped all consumer and producer ,then restarted
everything , recreated topics and now guess what? :)

one node reports... 
{code:java}

[2017-10-13 15:54:52,893] INFO Partition [__consumer_offsets,5] on broker 2: Expanding ISR
for partition __consumer_offsets-5 from 10,13,2 to 10,13,2,5 (kafka.cluster.Partition)
[2017-10-13 15:54:52,906] INFO Partition [__consumer_offsets,5] on broker 2: Cached zkVersion
[13] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
[2017-10-13 15:54:52,908] INFO Partition [__consumer_offsets,25] on broker 2: Expanding ISR
for partition __consumer_offsets-25 from 10,2,13 to 10,2,13,5 (kafka.cluster.Partition)
[2017-10-13 15:54:52,915] INFO Partition [__consumer_offsets,25] on broker 2: Cached zkVersion
[10] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
[2017-10-13 15:54:52,916] INFO Partition [__consumer_offsets,45] on broker 2: Expanding ISR
for partition __consumer_offsets-45 from 10,13,2 to 10,13,2,5 (kafka.cluster.Partition)
[2017-10-13 15:54:52,925] INFO Partition [__consumer_offsets,45] on broker 2: Cached zkVersion
[15] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
[2017-10-13 15:54:52,926] INFO Partition [__consumer_offsets,5] on broker 2: Expanding ISR
for partition __consumer_offsets-5 from 10,13,2 to 10,13,2,5 (kafka.cluster.Partition)
[2017-10-13 15:54:52,936] INFO Partition [__consumer_offsets,5] on broker 2: Cached zkVersion
[13] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
[2017-10-13 15:54:52,939] INFO Partition [__consumer_offsets,25] on broker 2: Expanding ISR
for partition __consumer_offsets-25 from 10,2,13 to 10,2,13,5 (kafka.cluster.Partition)
{code}

while others 


{code:java}
[2017-10-13 15:57:08,128] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,40]
to broker 2:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-10-13 15:57:09,129] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,40]
to broker 2:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-10-13 15:57:10,260] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,40]
to broker 2:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-10-13 15:57:11,262] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,40]
to broker 2:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-10-13 15:57:12,265] ERROR [ReplicaFetcherThread-0-2], Error for partition [__consumer_offsets,40]
to broker 2:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is
not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)
[2017-10-13 15:57:13,289] ERROR [ReplicaFetcherThread-0-2], Error fo
{code}


cluster still being inconsistent, I've also added 2 more nodes hoping in an increasing of
stability but nothing, I don't know if something is wrong because if kafka do some kind of
pre-flight checks during startup it does log nothing.. the only logs are those which have
no sense because the leader should be re-elected when there are ISR available.. and there
are 
I've started looking for an alternative software to  use, I'm trying to use kafka is so frustrating
:(


> Cached zkVersion not equal to that in zookeeper, broker not recovering.
> -----------------------------------------------------------------------
>
>                 Key: KAFKA-2729
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2729
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.8.2.1, 0.9.0.0, 0.10.0.0, 0.10.1.0, 0.11.0.0
>            Reporter: Danil Serdyuchenko
>
> After a small network wobble where zookeeper nodes couldn't reach each other, we started
seeing a large number of undereplicated partitions. The zookeeper cluster recovered, however
we continued to see a large number of undereplicated partitions. Two brokers in the kafka
cluster were showing this in the logs:
> {code}
> [2015-10-27 11:36:00,888] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker
5: Shrinking ISR for partition [__samza_checkpoint_event-creation_1,3] from 6,5 to 5 (kafka.cluster.Partition)
> [2015-10-27 11:36:00,891] INFO Partition [__samza_checkpoint_event-creation_1,3] on broker
5: Cached zkVersion [66] not equal to that in zookeeper, skip updating ISR (kafka.cluster.Partition)
> {code}
> For all of the topics on the effected brokers. Both brokers only recovered after a restart.
Our own investigation yielded nothing, I was hoping you could shed some light on this issue.
Possibly if it's related to: https://issues.apache.org/jira/browse/KAFKA-1382 , however we're
using 0.8.2.1.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message