Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
Received-SPF: unknown (nike.apache.org: error in processing during lookup of
 strib@nicira.com)
Message-ID: <4D446926.4090602@nicira.com>
Date: Sat, 29 Jan 2011 11:23:18 -0800
From: Jeremy Stribling <strib@nicira.com>
User-Agent: Thunderbird 2.0.0.24 (X11/20100623)
MIME-Version: 1.0
To: user@zookeeper.apache.org
Subject: hang in zookeeper_close() in the mt C client
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi everyone,

I use the multithreaded ZK C client library (3.3.2), and I'm seeing my 
application hang, and the only thread in it that's doing anything 
interesting is this one:

Thread 8 (Thread 5644):
#0  0x00007f5d7bb5bbe4 in __lll_lock_wait () from /lib/libpthread.so.0
#1  0x00007f5d7bb59ad0 in pthread_cond_broadcast@@GLIBC_2.3.2 () from 
/lib/libpthread.so.0
#2  0x00007f5d793628f6 in unlock_completion_list (l=0x32b4d68) at 
.../zookeeper/src/c/src/mt_adaptor.c:66
#3  0x00007f5d79354d4b in free_completions (zh=0x32b4c80, 
callCompletion=1, reason=-116) at .../zookeeper/src/c/src/zookeeper.c:1069
#4  0x00007f5d79355008 in cleanup_bufs (zh=0x32b4c80, callCompletion=1, 
rc=-116) at .../thirdparty/zookeeper/src/c/src/zookeeper.c:1125
#5  0x00007f5d79353200 in destroy (zh=0x32b4c80) at 
.../thirdparty/zookeeper/src/c/src/zookeeper.c:366
#6  0x00007f5d79358e0e in zookeeper_close (zh=0x32b4c80) at 
.../zookeeper/src/c/src/zookeeper.c:2326
#7  0x00007f5d79356d18 in api_epilog (zh=0x32b4c80, rc=0) at 
.../zookeeper/src/c/src/zookeeper.c:1661
#8  0x00007f5d79362f2f in adaptor_finish (zh=0x32b4c80) at 
.../zookeeper/src/c/src/mt_adaptor.c:205
#9  0x00007f5d79358c8c in zookeeper_close (zh=0x32b4c80) at 
.../zookeeper/src/c/src/zookeeper.c:2297
....

I've seen some threads online about how there's a race condition 
associated with zookeeper_close, where if you app is making a 
synchronous call at the same time using the closed zk_handle, there 
could be a hang.  However, my app makes no synchronous calls, and I'm 
99% sure that no other thread in my app is making any concurrent call 
into the library ('thread apply all bt' in gdb doesn't show any other 
usage of the library, anyway).

Has anyone seen this before?  Any leads?  Thanks,

Jeremy