zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahadev Konar <maha...@yahoo-inc.com>
Subject Re: What happens when a server loses all its state?
Date Wed, 17 Dec 2008 00:46:43 GMT
Hi Thomas,
 Here is what would happen in the scenario you mentioned.

> Great - thanks Mahadev.
> Not to drag this on more than necessary, please bear with me for one
> more example of 'amnesia' that comes to mind. I have a set of ZooKeeper
> servers A, B, C.
> - C is currently not running, A is the leader, B is the follower.
> - A proposes zxid1 to A and B, both acknowledge.
> - A asks A to commit (which it persists), but before the same commit
> request reaches B, all servers go down (say a power failure).
In this case, the zookeeper protocol says that zxid1 would be available only
if the client gets a success. So zxid1 may or may not get committed if A and
B come up later. ( this is a different scenario then what you mention

> - Later, B and C come up (A is slow to reboot), but B has lost all state
> due to disk failure.
This is how zookeeper would work in this scenario ---

Now since we have B and C come up and B has the most recent state but loses
it, then zookeeper is clueless about this. So C would say I have the some
zxid say zxid-n and B would say that I have zxid = 0 (since its stateless)
and C would become a leader (since it has the highest zxid).

This would lead to loss of data and loss of state in zookeeper. That's what
I meant when I mentioned that zookeeper relies heavily on the state being
persisted on disk.

> - C becomes the new leader and perhaps continues with some more new
> transactions.
Now if A comes back again, C would say that its the leader and ask A to
truncate all the transactions that A had to come to sync with C.

Again, you can see that how persistence loss can trigger state loss in
zookeeper. If its just minority of servers failing then this can be taken
care of by zookeeper but in this scenario is C failing and then being
brought up with an inconsisten state with another failure of A and data loss
of B -- which zookeeper cannot handle.

I hope this helps. 


On 12/16/08 4:02 PM, "Thomas Vinod Johnson" <Thomas.Johnson@Sun.COM> wrote:

> Mahadev Konar wrote:
>> Hi Thomas,
>>> More generally, is it a safe assumption to make that the ZooKeeper
>>> service will maintain all its guarantees if a minority of servers lose
>>> persistent state (due to bad disks, etc) and restart at some point in
>>> the future?
>> Yes that is true.
> Likely I'm misunderstanding the protocol, but have I effectively lost
> zxid1 at this point? What would happen when A comes back up?
> Thanks.

View raw message