zookeeper-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Nixon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ZOOKEEPER-261) Reinitialized servers should not participate in leader election
Date Tue, 06 Dec 2016 23:06:59 GMT

    [ https://issues.apache.org/jira/browse/ZOOKEEPER-261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727016#comment-15727016

Brian Nixon commented on ZOOKEEPER-261:

Ben and I discussed this offline. When starting up without any local data, the safest thing
to do is view this lack with extreme suspicion and not participate in voting until you can
pull down the data tree from the rest of the ensemble. Such a server is not qualified to confirm
which servers are up to date and could inadvertently elect a server that is missing some data.
The one exception is the creation of a fresh ensemble, when there is no data to repopulate
the local data tree. It's not clear that an ensemble can detect that it is in this state on
its own since in the worst case, every server will be subject to the same data losing fault
(in which case you should recover from backups instead of coming online as an empty data base).
This extra information needs to come from the admin.

With the changes from ZOOKEEPER-2325, a server with no local data tree starts with a zxid
of 0. I'll submit a pull request that changes that initial zxid to -1 unless a special 'initialize'
file is present in the data directory and removes voting privileges from members reporting
-1. The idea is that creating the 'initialize' file alongside 'myid' will be a standard part
of ensemble creation - the extra information from the admin. The 'initialize' file will be
automatically cleaned up by the server and subsequent restarts can view missing data directories
as a sign they are legitimately missing context (e.g. adding to an existing ensemble).

> Reinitialized servers should not participate in leader election
> ---------------------------------------------------------------
>                 Key: ZOOKEEPER-261
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-261
>             Project: ZooKeeper
>          Issue Type: Improvement
>          Components: leaderElection, quorum
>            Reporter: Benjamin Reed
> A server that has lost its data should not participate in leader election until it has
resynced with a leader. Our leader election algorithm and NEW_LEADER commit assumes that the
followers voting on a leader have not lost any of their data. We should have a flag in the
data directory saying whether or not the data is preserved so that the the flag will be cleared
if the data is ever cleared.
> Here is the problematic scenario: you have have ensemble of machines A, B, and C. C is
down. the last transaction seen by C is z. a transaction, z+1, is committed on A and B. Now
there is a power outage. B's data gets reinitialized. when power comes back up, B and C comes
up, but A does not. C will be elected leader and transaction z+1 is lost. (note, this can
happen even if all three machines are up and C just responds quickly. in that case C would
tell A to truncate z+1 from its log.) in theory we haven't violated our 2f+1 guarantee, since
A is failed and B still hasn't recovered from failure, but it would be nice if when we don't
have quorum that system stops working rather than works incorrectly if we lose quorum.

This message was sent by Atlassian JIRA

View raw message