hadoop-zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Flavio Junqueira (JIRA)" <j...@apache.org>
Subject [jira] Commented: (ZOOKEEPER-917) Leader election selected incorrect leader
Date Thu, 04 Nov 2010 11:17:44 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928179#action_12928179
] 

Flavio Junqueira commented on ZOOKEEPER-917:
--------------------------------------------

Hi Alexandre, It is an key premise of important replication algorithms, like Paxos, that there
is a portion of the state that persists across crashes (and recoveries). By replacing server
2 with a fresh server, you simply got rid of the persistent state. In general, making that
replacement you've made may lead you to trouble due to the problem I exposed a few postings
up. Of course, if you wait for a successful election, the problem is supposed to go away because
you have reestablished a quorum and this quorum does not contain the faulty server, but then
you have to make sure the election happens before you introduce the fresh server perhaps through
jmx or by inspecting the logs. Simply setting a reasonable timeout will work in most cases,
but the leader election is not guaranteed to succeed, and there is a chance, likely to be
small, that you'll end up with a corrupt state. 



> Leader election selected incorrect leader
> -----------------------------------------
>
>                 Key: ZOOKEEPER-917
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-917
>             Project: Zookeeper
>          Issue Type: Bug
>          Components: leaderElection, server
>    Affects Versions: 3.2.2
>         Environment: Cloudera distribution of zookeeper (patched to never cache DNS entries)
> Debian lenny
>            Reporter: Alexandre Hardy
>            Priority: Critical
>             Fix For: 3.3.3, 3.4.0
>
>         Attachments: zklogs-20101102144159SAST.tar.gz
>
>
> We had three nodes running zookeeper:
>   * 192.168.130.10
>   * 192.168.130.11
>   * 192.168.130.14
> 192.168.130.11 failed, and was replaced by a new node 192.168.130.13 (automated startup).
The new node had not participated in any zookeeper quorum previously. The node 192.148.130.11
was permanently removed from service and could not contribute to the quorum any further (powered
off).
> DNS entries were updated for the new node to allow all the zookeeper servers to find
the new node.
> The new node 192.168.130.13 was selected as the LEADER, despite the fact that it had
not seen the latest zxid.
> This particular problem has not been verified with later versions of zookeeper, and no
attempt has been made to reproduce this problem as yet.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message