zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@apache.org>
Subject Re: All nodes shutting down simultaneously
Date Sun, 24 Jan 2016 14:19:43 GMT
Hi Bruno,

If the leader crashes, then the followers are supposed to start complaining around the same
time. Was the server that powered down the leader of the ensemble?

Another leader election shouldn't take two minutes, so you could check the logs for what happened
around the time the ensemble wasn't able to get a leader established. Were election notifications
being exchanged regularly? Was the size of the snapshot too large and the init limit was kicking
in?

When you say that the servers shut themselves down, I suppose you're referring to the fact
that you've seen a message in the logs saying it. The server process remained up and they
haven't exited, yes?

-Flavio

> On 24 Jan 2016, at 12:05, Bruno Wassermann <baruch_w@hotmail.com> wrote:
> 
> Hi I have recently come across something I have not seen before: all 5 nodes of a ZK
cluster shutting themselves down at around the same time.  All servers worked fine for a long
time. Ranging from 2016-01-17 19:31 to 19:33:xx, the servers complained about not being able
to follow the leader and the zk processes shut themselves down. One of the servers (actual
host as in bare-metal) was somehow powered off at 19:33.  The ZK cluster was down for approximately
two minutes. Leader election took 90 seconds. The leader then expired more than 100 client
connections. A snippet from the log file of one of the nodes:2016-01-17 19:32:17,081 [myid:1]
- WARN  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@89] - Exception when following the
leaderjava.net.SocketTimeoutException: Read timed out    at java.net.SocketInputStream.socketRead0(Native
Method)    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)    at java.net.SocketInputStream.read(SocketInputStream.java:170)
   at java.net.SocketInputStream.read(SocketInputStream.java:141)    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
   at java.io.BufferedInputStream.read(BufferedInputStream.java:265)    at java.io.DataInputStream.readInt(DataInputStream.java:387)
   at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)    at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
   at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)    at org.apache.zookeeper.server.quorum.Learner.readPacket(Learner.java:153)
   at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:85)    at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:786)2016-01-17
19:32:17,081 [myid:1] - INFO  [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@166] - shutdown
calledjava.lang.Exception: shutdown Follower    at org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:166)
   at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:790)Followed by closing
client connections. I have analysed garbage collection behaviour on these five nodes and found
that on that day two of the nodes underwent one GC collection of 80 ms each.  What does the
read time out indicate? Does this mean that there was a network issue or was there maybe too
much data to sync on in a timely manner? Why would all nodes shut down at pretty much the
same time?   All the best,  -- Bruno 		 	   		  


Mime
View raw message