Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (athena.apache.org: domain of taylor.osmun@ca.ibm.com
 designates 32.97.182.138 as permitted sender)
Subject: 3.5.0-alpha-1615249: Issue with Zxid during Observer failover to
 Read-Only
To: user@zookeeper.apache.org
Message-ID: 
 <OF9CBED23B.79935A3D-ON85257DA0.007608AB-85257DA0.00761A22@ca.ibm.com>
From: Taylor Osmun <Taylor.Osmun@ca.ibm.com>
Date: Sun, 30 Nov 2014 16:29:59 -0500
MIME-Version: 1.0
Content-type: multipart/alternative;
	Boundary="0__=0ABBF733DFE58E3B8f9e8a93df938690918c0ABBF733DFE58E3B"
Content-Disposition: inline

--0__=0ABBF733DFE58E3B8f9e8a93df938690918c0ABBF733DFE58E3B
Content-type: text/plain; charset=US-ASCII


I currently have the following configuration:
 * 1 server running locally
 * 1 observer running locally (which accepts read-only mode)

1) I begin by starting up the server, and then the observer, both operate
normally and Zxid is in sync:
   echo stat | nc localhost 2181
   Zookeeper version: 3.5.0-alpha-1615249, built on 08/01/2014 22:13 GMT
   Clients:
    /127.0.0.1:45432[0](queued=0,recved=1,sent=0)

   Latency min/avg/max: 0/0/0
   Received: 2
   Sent: 1
   Connections: 1
   Outstanding: 0
   Zxid: 0x100000003
   Mode: leader
   Node count: 6

   echo stat | nc localhost 2182
   Zookeeper version: 3.5.0-alpha-1615249, built on 08/01/2014 22:13 GMT
   Clients:
    /127.0.0.1:36159[0](queued=0,recved=1,sent=0)

   Latency min/avg/max: 9/44/112
   Received: 6
   Sent: 5
   Connections: 1
   Outstanding: 0
   Zxid: 0x100000003
   Mode: observer
   Node count: 6
2) However, when I shutdown the server, the observer properly enters read
only mode, but resets its Zxid to 0. It does not reload the database:
   echo stat | nc localhost 2182
   Zookeeper version: 3.5.0-alpha-1615249, built on 08/01/2014 22:13 GMT
   READ-ONLY mode; serving only read-only clients
   Clients:
    /127.0.0.1:36179[0](queued=0,recved=1,sent=0)

   Latency min/avg/max: 0/0/0
   Received: 3
   Sent: 2
   Connections: 1
   Outstanding: 0
   Zxid: 0x0
   Mode: read-only
   Node count: 5

As a result, my read-only observer instance is essentially empty. The only
way to fix this is to restart the observer, which properly reads the local
database.
The issue I can see here (according to source) is that the ZKDatabase is
not reloaded when the ReadOnlyZooKeeperServer is initialized (after
ZKDatabase has been cleared).
Thoughts? Is this known?

Note that restarting the read-only observer results in the proper Zxid
being used, as it loads the database properly.

Cheers,
Taylor
--0__=0ABBF733DFE58E3B8f9e8a93df938690918c0ABBF733DFE58E3B--