kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jun Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3042) updateIsr should stop after failed several times due to zkVersion issue
Date Tue, 19 Apr 2016 23:52:25 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15248954#comment-15248954

Jun Rao commented on KAFKA-3042:

Currently, the main differences between LeaderAndIsrRequest and UpdateMetadataRequest are
that (1) LeaderAndIsrRequest is only sent to brokers storing the partition while UpdateMetadataRequest
is sent to every broker; (2) LeaderAndIsrRequest only includes the port used by interBrokerSecurityProtocol
while UpdateMetadataRequest includes all ports. 

One of the original intentions of the separation is probably to save the amount of data communication,
which may no longer be valid. It's probably simpler to combine LeaderAndIsrRequest and UpdateMetadataRequest
into one request. However, we have to look at the code more carefully to understand the full
implication. For example, in ReplicaManager, we have the logic that relies upon the very first
LeaderAndIsrRequest to initialize the partition list for checkpointing the high watermarks.

To fix this particular issue, the simplest approach is to send UpdateMetadataRequest first
during controller failover. It shouldn't do any harm since it just communicates the current
state from ZK to all brokers.

It would be useful to fix a couple of other minor issues in this jira too.
1, Log liveBrokers in LeaderAndIsrRequest and UpdateMetadataRequest in state-change log.
2. LeaderAndIsrRequest has a controllerEpoch at the topic level and a controllerEpoch per
partition. The former indicates the epoch of the controller that sends the LeaderAndIsrRequest.
The latter indicates the epoch of the controller that last changed the leader/isr for that
partition. When logging LeaderAndIsrRequest in state-change log in ReplicaManager, we mixed
the two up in some of the cases. For example, in makeLeaders(), we log the top level controllerEpoch.
In makeFollowers(), we log the controllerEpoch at the partition level. We should make the
logging clearer and consistent.

> updateIsr should stop after failed several times due to zkVersion issue
> -----------------------------------------------------------------------
>                 Key: KAFKA-3042
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3042
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions:
>         Environment: jdk 1.7
> centos 6.4
>            Reporter: Jiahongchao
>             Fix For:
>         Attachments: controller.log, server.log.2016-03-23-01, state-change.log
> sometimes one broker may repeatly log
> "Cached zkVersion 54 not equal to that in zookeeper, skip updating ISR"
> I think this is because the broker consider itself as the leader in fact it's a follower.
> So after several failed tries, it need to find out who is the leader

This message was sent by Atlassian JIRA

View raw message