Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: pass (nike.apache.org: domain of briantarbox@gmail.com
 designates 209.85.217.181 as permitted sender)
MIME-Version: 1.0
Date: Thu, 23 May 2013 09:21:06 -0400
Message-ID: 
 <CAP8NC12-5VfW8FJ91pjbmmrWF7hppBBAf5LnRCEXjY1Us2yTWA@mail.gmail.com>
Subject: cluster confused...would not delete node until cluster restarted...
From: Brian Tarbox <briantarbox@gmail.com>
To: user@zookeeper.apache.org
Content-Type: multipart/alternative; boundary=089e0158c5260a04c504dd628ebb

--089e0158c5260a04c504dd628ebb
Content-Type: text/plain; charset=ISO-8859-1

My 3 node cluster would not let me delete a node saying it was not
empty...but stat showed that it was in fact empty:

[zk: localhost:2181(CONNECTED) 1] delete /ROOT_A/INSTANCES/
10.244.43.240/WORKERS
*Node not empty:* /ROOT_A/INSTANCES/10.244.43.240/WORKERS

[zk: localhost:2181(CONNECTED) 0] stat /ROOT_A/INSTANCES/
10.244.43.240/WORKERS
cZxid = 0xe0015ad31
ctime = Wed May 22 17:20:52 EDT 2013
mZxid = 0xe0015ad31
mtime = Wed May 22 17:20:52 EDT 2013
pZxid = 0xe0015ae3c
cversion = 2
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 24
*numChildren = 0*
*
*
*
*
The debug log showed this on the machine initiating the delete:
2013-05-23 09:05:37,030 [myid:1] - DEBUG
[FollowerRequestProcessor:1:CommitProcessor@171] - Processing request::
sessionid:0x13ed17dc4310000 type:delete cxid:0x3 zxid:0xfffffffffffffffe
txntype:unknown reqpath:/ROOT_A/INSTANCES/10.244.43.240/WORKERS

2013-05-23 09:05:37,034 [myid:1] - DEBUG
[QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:CommitProcessor@161] - Committing
request:: sessionid:0x13ed17dc4310000 type:error cxid:0x3 zxid:0xe00200606
txntype:-1 reqpath:n/a

2013-05-23 09:05:37,034 [myid:1] - DEBUG
[CommitProcessor:1:FinalRequestProcessor@88] - Processing request::
sessionid:0x13ed17dc4310000 type:delete cxid:0x3 zxid:0xe00200606
txntype:-1 reqpath:/ROOT_A/INSTANCES/10.244.43.240/WORKERS
2013-05-23 09:05:37,034 [myid:1] - DEBUG [CommitProcessor:1:DataTree@949] -
*Ignoring processTxn failure hdr: -1 : error: -111*
*
*
And on another node I saw this:
2013-05-23 09:11:22,373 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@627] - Got user-level KeeperException when
processing sessionid:0x13ed18295a60000 type:delete cxid:0x2
zxid:0xe0020060b txntype:-1 reqpath:n/a Error Path:/ROOT_A/INSTANCES/
10.244.43.240/WORKERS Error:KeeperErrorCode = Directory not empty for
/ROOT_A/INSTANCES/10.244.43.240/WORKERS*
*

Third node say nothing in response to the delete.

Problem went away after I serially restarted the first two machines so I'm
left with a "working" system and an uneasy feeling.  Was there anything I
could have done other than restart the servers?

Thanks.
-- 
http://about.me/BrianTarbox

--089e0158c5260a04c504dd628ebb--