zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject ZooKeeperServer#shutdown hangs
Date Wed, 16 Dec 2015 19:46:23 GMT
Hi,
HBase recently upgraded to zookeeper 3.4.7

In one of the tests, TestSplitLogManager, there is reproducible hang at the
end of the test.
Below is snippet from stack trace related to zookeeper:

"main-EventThread" daemon prio=5 tid=0x00007fd27488a800 nid=0x6f1f waiting
on condition [0x000000011834b000]
   java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x00000007c5b8d3a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"main-SendThread(localhost:59510)" daemon prio=5 tid=0x00007fd274eb4000
nid=0x9513 waiting on condition [0x0000000118042000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
  at java.lang.Thread.sleep(Native Method)
  at
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
  at
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
  at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)

"SyncThread:0" prio=5 tid=0x00007fd274d02000 nid=0x730f waiting for monitor
entry [0x00000001170ac000]
   java.lang.Thread.State: BLOCKED (on object monitor)
  at
org.apache.zookeeper.server.ZooKeeperServer.decInProcess(ZooKeeperServer.java:512)
  - waiting to lock <0x00000007c5b62128> (a
org.apache.zookeeper.server.ZooKeeperServer)
  at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:144)
  at
org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
  at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)

"main-EventThread" daemon prio=5 tid=0x00007fd2753a3800 nid=0x711b waiting
on condition [0x0000000117a30000]
   java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x00000007c9b106b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
  at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
  at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
  at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)

"main" prio=5 tid=0x00007fd276000000 nid=0x1903 in Object.wait()
[0x0000000108aa1000]
   java.lang.Thread.State: WAITING (on object monitor)
  at java.lang.Object.wait(Native Method)
  - waiting on <*0x00000007c5b66400*> (a
org.apache.zookeeper.server.SyncRequestProcessor)
  at java.lang.Thread.join(Thread.java:1281)
  - locked <*0x00000007c5b66400*> (a
org.apache.zookeeper.server.SyncRequestProcessor)
  at java.lang.Thread.join(Thread.java:1355)
  at
org.apache.zookeeper.server.SyncRequestProcessor.shutdown(SyncRequestProcessor.java:213)
  at
org.apache.zookeeper.server.PrepRequestProcessor.shutdown(PrepRequestProcessor.java:770)
  at
org.apache.zookeeper.server.ZooKeeperServer.shutdown(ZooKeeperServer.java:478)
  - locked <0x00000007c5b62128> (a
org.apache.zookeeper.server.ZooKeeperServer)
  at
org.apache.zookeeper.server.NIOServerCnxnFactory.shutdown(NIOServerCnxnFactory.java:266)
  at
org.apache.hadoop.hbase.zookeeper.MiniZooKeeperCluster.shutdown(MiniZooKeeperCluster.java:301)

Note the bold address in the last hunk which seems to indicate some form of
deadlock.

I can send the full stack trace upon request.
When reverting to 3.4.6, the test passes.

Comment / hint is welcome.

Cheers

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message