zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Heffner <m...@librato.com>
Subject Recovering from zxid rollover
Date Tue, 23 May 2017 22:47:26 GMT
Hi,

I'm curious what the best practices are for handling zxid rollover in a ZK
ensemble. We have a few five-node ZK ensembles (some 3.4.8 and some 3.3.6)
and they periodically rollover their zxid. We see the following in the
system logs on the leader node:

2017-05-22 12:54:14,117 [myid:15] - ERROR [ProcessThread(sid:15
cport:-1)::ZooKeeperCriticalThread@49] - Severe unrecoverable error, from
thread : ProcessThread(sid:15 cport:-1):
org.apache.zookeeper.server.RequestProcessor$RequestProcessorException:
zxid lower 32 bits have rolled over, forcing re-election, and therefore new
epoch start

>From my best understanding of the code, this exception will end up causing
the leader to enter shutdown():

https://github.com/apache/zookeeper/blob/09cd5db55446a4b390f82e3548b929f19e33430d/src/java/main/org/apache/zookeeper/server/ZooKeeperServer.java#L464-L464

This shuts down the zookeeper instance from servicing requests, but the JVM
is still actually running. What we experience is that while this ZK
instance is still running, the remaining follower nodes can't re-elect a
leader (at least within 15 mins) and quorum is offline. Our remediation so
far has been to restart the original leader node, at which point the
cluster recovers.

The two questions I have are:

1. Should the remaining 4 nodes be able to re-elect a leader after zxid
rollover without intervention (restarting)?

2. If the leader enters shutdown() state after a zxid rollover, is there
any scenario where it will return to started? If not, how are others
handling this scenario -- maybe a healthcheck that kills/restarts an
instance that is in shutdown state?


Cheers,

Mike


-- 

  Mike Heffner <mike@librato.com>
  Librato, Inc.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message