zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh R <rake...@huawei.com>
Subject RE: Zookeeper Cluster with quorum Lost, what next ?
Date Mon, 28 Oct 2013 07:22:52 GMT
Hi S├ębastien,

>>>>>In case the quorum is lost in a Zookeeper Cluster of 2 * n + 1 nodes (so
more than n server died, or only one server is still alive for example), what happen next
:

Here the all live servers will be participating to form quorum by doing leader election and
will be in LOOKING state.

>>>>>- do other alive servers of the cluster continue to respond to all request
of clients (read request and write request) ?
>>>>- do other alive servers of the cluster continue to respond but only to read
request of clients (write request are all rejected) ?
>>>>- do other alive servers reject all requests from client ?
For performing write/read operation, it requires quorum(LEADER and FOLLOWER's). In your case,
it has 2 * n + 1 failures, all the live servers will be participating in quorum formation
and in LOOKING state. All the existing clients would get notified with DISCONNECTED event,
also any read/write client requests will be rejected and you can see connection loss exceptions
in the client side.

If you want to serve read client request(using alive servers) after quorum partition or fails,
there is another good feature in ZooKeeper - read-only server. 
In this case, the live servers will transition to read-only mode and will serve the read requests
from read-only clients. Also, in background all these servers will be participating in leader
election to form quorum.
Please refer 'Read Only Mode Server' section in http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#Experimental+Options%2FFeatures

>>>>- do other alive servers die ?
No, all the live servers would continuously tries to form quorum and would be in leader election
phase. Now the Server state will be LOOKING.
Please refer http://zookeeper.apache.org/doc/r3.2.2/zookeeperInternals.html#sc_leaderElection

>>>>- do other alive servers reject connection from client that were connected
to a server that died ?
>>>>- do other alive servers accept new connection from client (wether or not
these client were connected with a session to an other serveer that died) ?
>>>>- do other alive servers disconnect all client ?
Existing Clients will get DISCONNECTED event and will infinitely retries connecting to all
the servers mentioned in the connect string. New client establishment also will fail and client
will continue to retries infinitely. 


In general, say ZooKeeper quorum losts: 
	- Live servers will be in leader election phase and rejects all the client connections.
	- Existing clients will get DISCONNECTED event and will infinitely retries to all the servers
mentioned in the connect string.
	http://zookeeper.apache.org/doc/r3.1.2/zookeeperProgrammers.html#ch_zkSessions


There is a special feature addition of read-only server to serve read-only client requests
even after quorum lost. Please refer following link to understand more on this:
https://issues.apache.org/jira/i#browse/ZOOKEEPER-784
http://zookeeper.apache.org/doc/trunk/zookeeperAdmin.html#Experimental+Options%2FFeatures

Hope these information will be helpful.

-Rakesh

-----Original Message-----
From: Seb Geek [mailto:geekseb@gmail.com] 
Sent: 27 October 2013 23:09
To: user@zookeeper.apache.org
Subject: Zookeeper Cluster with quorum Lost, what next ?

Hello,

My Context : I want to create a cluster of 2 * n + 1 nodes zookeeper servers and several java
clients (like Solr Cloud nodes or homemade java client that implement and use receipt for
Zookeeper).

In case the quorum is lost in a Zookeeper Cluster of 2 * n + 1 nodes (so more than n server
died, or only one server is still alive for example), what happen next :

- do other alive servers of the cluster continue to respond to all request of clients (read
request and write request) ?
- do other alive servers of the cluster continue to respond but only to read request of clients
(write request are all rejected) ?
- do other alive servers reject all requests from client ?
- do other alive servers die ?

- do other alive servers reject connection from client that were connected to a server that
died ?
- do other alive servers accept new connection from client (wether or not these client were
connected with a session to an other serveer that died) ?
- do other alive servers disconnect all client ?


And my last question : What happen next when the quorum is restored (died servers are restarted)
? are pending write requests applied in the zookeeper cluster ? ...

Thank you in advance for your answers,

Regards,
S├ębastien
Mime
View raw message