zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Laurie Turner (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-2164) fast leader election keeps failing
Date Mon, 16 Oct 2017 01:31:03 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205360#comment-16205360
] 

Laurie Turner commented on ZOOKEEPER-2164:
------------------------------------------

I believe I have run into this issue (zookeeper versions 3.4.6  and 3.4.10).  

These scenarios I've tested  lead me to believe I have the same problem.  I have a 3 node
cluster and if the leader is "2" and is stopped, the election will fail and ultimately 1 and
3  respond with  "This ZooKeeper instance is not currently serving requests" from the stat
command.  

If 2 is restarted, the cluster returns and 2 becomes the leader .     This appears to be the
scenario documented above.  Sometimes 3 will fail to rejoin but if it is restarted it will
rejoin the cluster successfully.

Essentially the only electable leader is #2.  The nodes are built as docker containers and
orchestrated using Kubernetes.  

I am searching  for a work around or  configuration change that will enable the  cluster to
be functional if the existing leader fails  are there are only 2 nodes (out of 3) available.

> fast leader election keeps failing
> ----------------------------------
>
>                 Key: ZOOKEEPER-2164
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2164
>             Project: ZooKeeper
>          Issue Type: Bug
>          Components: leaderElection
>    Affects Versions: 3.4.5
>            Reporter: Michi Mutsuzaki
>             Fix For: 3.5.4, 3.6.0
>
>
> I have a 3-node cluster with sids 1, 2 and 3. Originally 2 is the leader. When I shut
down 2, 1 and 3 keep going back to leader election. Here is what seems to be happening.
> - Both 1 and 3 elect 3 as the leader.
> - 1 receives votes from 3 and itself, and starts trying to connect to 3 as a follower.
> - 3 doesn't receive votes for 5 seconds because connectOne() to 2 doesn't timeout for
5 seconds: https://github.com/apache/zookeeper/blob/41c9fcb3ca09cd3d05e59fe47f08ecf0b85532c8/src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java#L346
> - By the time 3 receives votes, 1 has given up trying to connect to 3: https://github.com/apache/zookeeper/blob/41c9fcb3ca09cd3d05e59fe47f08ecf0b85532c8/src/java/main/org/apache/zookeeper/server/quorum/Learner.java#L247
> I'm using 3.4.5, but it looks like this part of the code hasn't changed for a while,
so I'm guessing later versions have the same issue.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message