hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Date Wed, 02 Nov 2011 15:39:32 GMT

     [ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-4724:
-------------------------

      Description: 
fom the logs in my env
{noformat}
2011-11-01 15:48:40,744 WARN  [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
{noformat}
Anyway, after this the logs finishes with:
{noformat}
2011-11-01 15:54:35,132 INFO  [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
{noformat}
it's in
{noformat}
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
    org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
    org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
    org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
    org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
    org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
    org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
{noformat}
So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.

I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
{noformat}
Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
  State: TIMED_WAITING
  Blocked count: 360
  Waited count: 359
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
  State: WAITING
  Blocked count: 0
  Waited count: 4
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 152 (Master:0;localhost,39664,1320187706355):
  State: WAITING
  Blocked count: 217
  Waited count: 174
  Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
    org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
    org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
    org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
    org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
    org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
    java.lang.Thread.run(Thread.java:662)
Thread 165 (LruBlockCache.EvictionThread):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
    java.lang.Thread.run(Thread.java:662)
Thread 151 (main-EventThread):
  State: WAITING
  Blocked count: 40
  Waited count: 39
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 150 (main-SendThread(localhost:21819)):
  State: RUNNABLE
  Blocked count: 55
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 149 (IPC Server handler 4 on 39664):
  State: WAITING
  Blocked count: 3
  Waited count: 1968
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 148 (IPC Server handler 3 on 39664):
  State: WAITING
  Blocked count: 7
  Waited count: 1969
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 147 (IPC Server handler 2 on 39664):
  State: WAITING
  Blocked count: 11
  Waited count: 1968
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 146 (IPC Server handler 1 on 39664):
  State: WAITING
  Blocked count: 6
  Waited count: 1969
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 145 (IPC Server handler 0 on 39664):
  State: WAITING
  Blocked count: 3
  Waited count: 1970
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 133 (IPC Server listener on 39664):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
Thread 144 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
Thread 143 (IPC Reader 9 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 142 (IPC Reader 8 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 141 (IPC Reader 7 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 140 (IPC Reader 6 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 139 (IPC Reader 5 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 138 (IPC Reader 4 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 137 (IPC Reader 3 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 136 (IPC Reader 2 on port 39664):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 135 (IPC Reader 1 on port 39664):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 134 (IPC Reader 0 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 124 (LeaseChecker):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 407
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
    java.lang.Thread.run(Thread.java:662)
Thread 121 (ProcessThread:-1):
  State: WAITING
  Blocked count: 0
  Waited count: 269
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
Thread 120 (SyncThread:0):
  State: WAITING
  Blocked count: 2
  Waited count: 252
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
Thread 119 (SessionTracker):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 207
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
  State: RUNNABLE
  Blocked count: 38
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 407
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
    java.lang.Thread.run(Thread.java:662)
Thread 113 (IPC Server handler 2 on 39899):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 112 (IPC Server handler 1 on 39899):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 111 (IPC Server handler 0 on 39899):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 106 (IPC Server listener on 39899):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 108 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
    sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
    org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
    java.lang.Thread.run(Thread.java:662)
Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
  State: TIMED_WAITING
  Blocked count: 137
  Waited count: 283
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
    java.lang.Thread.run(Thread.java:662)
Thread 107 (pool-4-thread-1):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 105 (Timer-3):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 15
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 103 (1185716172@qtp-1197660496-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 8
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 408
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
    java.lang.Thread.run(Thread.java:662)
Thread 86 (IPC Server handler 2 on 56098):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 85 (IPC Server handler 1 on 56098):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 84 (IPC Server handler 0 on 56098):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 80 (IPC Server listener on 56098):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 82 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
    sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
    org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
    java.lang.Thread.run(Thread.java:662)
Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
  State: TIMED_WAITING
  Blocked count: 149
  Waited count: 290
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
    java.lang.Thread.run(Thread.java:662)
Thread 81 (pool-3-thread-1):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 79 (Timer-2):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 15
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 77 (1905729203@qtp-164967086-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 8
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 408
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
    java.lang.Thread.run(Thread.java:662)
Thread 60 (IPC Server handler 2 on 40388):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 59 (IPC Server handler 1 on 40388):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 58 (IPC Server handler 0 on 40388):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 53 (IPC Server listener on 40388):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 55 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
    sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
    org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
    java.lang.Thread.run(Thread.java:662)
Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
  State: TIMED_WAITING
  Blocked count: 148
  Waited count: 283
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
    java.lang.Thread.run(Thread.java:662)
Thread 54 (pool-2-thread-1):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 52 (Timer-1):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 15
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 50 (1191055321@qtp-1068922666-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 8
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 31 (IPC Server handler 9 on 52227):
  State: WAITING
  Blocked count: 0
  Waited count: 64
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 30 (IPC Server handler 8 on 52227):
  State: WAITING
  Blocked count: 3
  Waited count: 65
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 29 (IPC Server handler 7 on 52227):
  State: WAITING
  Blocked count: 2
  Waited count: 66
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 28 (IPC Server handler 6 on 52227):
  State: WAITING
  Blocked count: 1
  Waited count: 64
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 27 (IPC Server handler 5 on 52227):
  State: WAITING
  Blocked count: 4
  Waited count: 68
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 26 (IPC Server handler 4 on 52227):
  State: WAITING
  Blocked count: 0
  Waited count: 64
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 25 (IPC Server handler 3 on 52227):
  State: WAITING
  Blocked count: 0
  Waited count: 65
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 24 (IPC Server handler 2 on 52227):
  State: WAITING
  Blocked count: 1
  Waited count: 66
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 23 (IPC Server handler 1 on 52227):
  State: WAITING
  Blocked count: 0
  Waited count: 64
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 22 (IPC Server handler 0 on 52227):
  State: WAITING
  Blocked count: 3
  Waited count: 65
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 15 (IPC Server listener on 52227):
  State: RUNNABLE
  Blocked count: 43
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 17 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 21 (Timer-0):
  State: TIMED_WAITING
  Blocked count: 1
  Waited count: 17
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 19 (1821050251@qtp-589734556-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 8
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 16 (pool-1-thread-1):
  State: RUNNABLE
  Blocked count: 45
  Waited count: 44
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
  State: TIMED_WAITING
  Blocked count: 1
  Waited count: 147
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
    java.lang.Thread.run(Thread.java:662)
Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
  State: TIMED_WAITING
  Blocked count: 2
  Waited count: 147
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
    java.lang.Thread.run(Thread.java:662)
Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 220
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
    java.lang.Thread.run(Thread.java:662)
Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 89
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
    java.lang.Thread.run(Thread.java:662)
Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 2
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
    java.lang.Thread.run(Thread.java:662)
Thread 4 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 9
  Waited count: 10
  Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 11
  Waited count: 12
  Waiting on java.lang.ref.Reference$Lock@270d75a3
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
  State: RUNNABLE
  Blocked count: 104
  Waited count: 97
  Stack:
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
    org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
    org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
    org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
    org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
    org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
    org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
    org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
{noformat}

  was:

fom the logs in my env
{noformat}
2011-11-01 15:48:40,744 WARN  [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
	at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
{noformat}
Anyway, after this the logs finishes with:
{noformat}
2011-11-01 15:54:35,132 INFO  [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
{noformat}
it's in
{noformat}
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
    org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
    org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
    org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
    org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
    org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
    org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
{noformat}
So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.

I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
{noformat}
Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
  State: TIMED_WAITING
  Blocked count: 360
  Waited count: 359
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
  State: WAITING
  Blocked count: 0
  Waited count: 4
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 152 (Master:0;localhost,39664,1320187706355):
  State: WAITING
  Blocked count: 217
  Waited count: 174
  Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
    org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
    org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
    org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
    org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
    org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
    java.lang.Thread.run(Thread.java:662)
Thread 165 (LruBlockCache.EvictionThread):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
    java.lang.Thread.run(Thread.java:662)
Thread 151 (main-EventThread):
  State: WAITING
  Blocked count: 40
  Waited count: 39
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
Thread 150 (main-SendThread(localhost:21819)):
  State: RUNNABLE
  Blocked count: 55
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
Thread 149 (IPC Server handler 4 on 39664):
  State: WAITING
  Blocked count: 3
  Waited count: 1968
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 148 (IPC Server handler 3 on 39664):
  State: WAITING
  Blocked count: 7
  Waited count: 1969
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 147 (IPC Server handler 2 on 39664):
  State: WAITING
  Blocked count: 11
  Waited count: 1968
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 146 (IPC Server handler 1 on 39664):
  State: WAITING
  Blocked count: 6
  Waited count: 1969
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 145 (IPC Server handler 0 on 39664):
  State: WAITING
  Blocked count: 3
  Waited count: 1970
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
Thread 133 (IPC Server listener on 39664):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
Thread 144 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
Thread 143 (IPC Reader 9 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 142 (IPC Reader 8 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 141 (IPC Reader 7 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 140 (IPC Reader 6 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 139 (IPC Reader 5 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 138 (IPC Reader 4 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 137 (IPC Reader 3 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 136 (IPC Reader 2 on port 39664):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 135 (IPC Reader 1 on port 39664):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 134 (IPC Reader 0 on port 39664):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 124 (LeaseChecker):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 407
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
    java.lang.Thread.run(Thread.java:662)
Thread 121 (ProcessThread:-1):
  State: WAITING
  Blocked count: 0
  Waited count: 269
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
Thread 120 (SyncThread:0):
  State: WAITING
  Blocked count: 2
  Waited count: 252
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
Thread 119 (SessionTracker):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 207
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
  State: RUNNABLE
  Blocked count: 38
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 407
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
    java.lang.Thread.run(Thread.java:662)
Thread 113 (IPC Server handler 2 on 39899):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 112 (IPC Server handler 1 on 39899):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 111 (IPC Server handler 0 on 39899):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 106 (IPC Server listener on 39899):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 108 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
    sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
    org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
    java.lang.Thread.run(Thread.java:662)
Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
  State: TIMED_WAITING
  Blocked count: 137
  Waited count: 283
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
    java.lang.Thread.run(Thread.java:662)
Thread 107 (pool-4-thread-1):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 105 (Timer-3):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 15
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 103 (1185716172@qtp-1197660496-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 8
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 408
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
    java.lang.Thread.run(Thread.java:662)
Thread 86 (IPC Server handler 2 on 56098):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 85 (IPC Server handler 1 on 56098):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 84 (IPC Server handler 0 on 56098):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 80 (IPC Server listener on 56098):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 82 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
    sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
    org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
    java.lang.Thread.run(Thread.java:662)
Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
  State: TIMED_WAITING
  Blocked count: 149
  Waited count: 290
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
    java.lang.Thread.run(Thread.java:662)
Thread 81 (pool-3-thread-1):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 79 (Timer-2):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 15
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 77 (1905729203@qtp-164967086-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 8
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 408
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
    java.lang.Thread.run(Thread.java:662)
Thread 60 (IPC Server handler 2 on 40388):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 59 (IPC Server handler 1 on 40388):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 58 (IPC Server handler 0 on 40388):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 53 (IPC Server listener on 40388):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 55 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
    sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
    sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
    org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
    java.lang.Thread.run(Thread.java:662)
Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
  State: TIMED_WAITING
  Blocked count: 148
  Waited count: 283
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
    org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
    java.lang.Thread.run(Thread.java:662)
Thread 54 (pool-2-thread-1):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 52 (Timer-1):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 15
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 50 (1191055321@qtp-1068922666-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 8
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
    java.lang.Thread.run(Thread.java:662)
Thread 31 (IPC Server handler 9 on 52227):
  State: WAITING
  Blocked count: 0
  Waited count: 64
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 30 (IPC Server handler 8 on 52227):
  State: WAITING
  Blocked count: 3
  Waited count: 65
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 29 (IPC Server handler 7 on 52227):
  State: WAITING
  Blocked count: 2
  Waited count: 66
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 28 (IPC Server handler 6 on 52227):
  State: WAITING
  Blocked count: 1
  Waited count: 64
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 27 (IPC Server handler 5 on 52227):
  State: WAITING
  Blocked count: 4
  Waited count: 68
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 26 (IPC Server handler 4 on 52227):
  State: WAITING
  Blocked count: 0
  Waited count: 64
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 25 (IPC Server handler 3 on 52227):
  State: WAITING
  Blocked count: 0
  Waited count: 65
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 24 (IPC Server handler 2 on 52227):
  State: WAITING
  Blocked count: 1
  Waited count: 66
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 23 (IPC Server handler 1 on 52227):
  State: WAITING
  Blocked count: 0
  Waited count: 64
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 22 (IPC Server handler 0 on 52227):
  State: WAITING
  Blocked count: 3
  Waited count: 65
  Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
    java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
Thread 15 (IPC Server listener on 52227):
  State: RUNNABLE
  Blocked count: 43
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
Thread 17 (IPC Server Responder):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
Thread 21 (Timer-0):
  State: TIMED_WAITING
  Blocked count: 1
  Waited count: 17
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:509)
    java.util.TimerThread.run(Timer.java:462)
Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
  State: RUNNABLE
  Blocked count: 2
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
    org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 19 (1821050251@qtp-589734556-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 8
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
Thread 16 (pool-1-thread-1):
  State: RUNNABLE
  Blocked count: 45
  Waited count: 44
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    java.lang.Thread.run(Thread.java:662)
Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
  State: TIMED_WAITING
  Blocked count: 1
  Waited count: 147
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
    java.lang.Thread.run(Thread.java:662)
Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
  State: TIMED_WAITING
  Blocked count: 2
  Waited count: 147
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
    java.lang.Thread.run(Thread.java:662)
Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 220
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
    java.lang.Thread.run(Thread.java:662)
Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 89
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
    java.lang.Thread.run(Thread.java:662)
Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 2
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
    java.lang.Thread.run(Thread.java:662)
Thread 4 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 9
  Waited count: 10
  Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 11
  Waited count: 12
  Waiting on java.lang.ref.Reference$Lock@270d75a3
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
  State: RUNNABLE
  Blocked count: 104
  Waited count: 97
  Stack:
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
    org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
    org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
    org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
    org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
    org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
    org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    java.lang.reflect.Method.invoke(Method.java:597)
    org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
    org.junit.runners.ParentRunner.run(ParentRunner.java:236)
    org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
{noformat}

    Fix Version/s: 0.92.0
    
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
>                 Key: HBASE-4724
>                 URL: https://issues.apache.org/jira/browse/HBASE-4724
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.94.0
>            Reporter: nkeywal
>            Assignee: nkeywal
>            Priority: Critical
>             Fix For: 0.92.0
>
>         Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN  [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> 	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO  [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
>     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
>     org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>     org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
>     org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
>     org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
>     org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
>     org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
>     org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
>   State: TIMED_WAITING
>   Blocked count: 360
>   Waited count: 359
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
>     org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 4
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
>   State: WAITING
>   Blocked count: 217
>   Waited count: 174
>   Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Object.wait(Object.java:485)
>     org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
>     org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
>     org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
>     org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
>     org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
>     org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
>     java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Object.wait(Object.java:485)
>     org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
>     java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
>   State: WAITING
>   Blocked count: 40
>   Waited count: 39
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
>   State: RUNNABLE
>   Blocked count: 55
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
>   State: WAITING
>   Blocked count: 3
>   Waited count: 1968
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
>   State: WAITING
>   Blocked count: 7
>   Waited count: 1969
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
>   State: WAITING
>   Blocked count: 11
>   Waited count: 1968
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
>   State: WAITING
>   Blocked count: 6
>   Waited count: 1969
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
>   State: WAITING
>   Blocked count: 3
>   Waited count: 1970
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 407
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
>     java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 269
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 252
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 207
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
>   State: RUNNABLE
>   Blocked count: 38
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 407
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
>     java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>     sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
>     sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>     org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
>     java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
>   State: TIMED_WAITING
>   Blocked count: 137
>   Waited count: 283
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
>     java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 15
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:509)
>     java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>     org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 8
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 408
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
>     java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>     sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
>     sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>     org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
>     java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
>   State: TIMED_WAITING
>   Blocked count: 149
>   Waited count: 290
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
>     java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 15
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:509)
>     java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>     org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 8
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 408
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
>     java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>     sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
>     sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>     org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
>     java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
>   State: TIMED_WAITING
>   Blocked count: 148
>   Waited count: 283
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
>     java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 15
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:509)
>     java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>     org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 8
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 64
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
>   State: WAITING
>   Blocked count: 3
>   Waited count: 65
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 66
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
>   State: WAITING
>   Blocked count: 1
>   Waited count: 64
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
>   State: WAITING
>   Blocked count: 4
>   Waited count: 68
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 64
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 65
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
>   State: WAITING
>   Blocked count: 1
>   Waited count: 66
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 64
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
>   State: WAITING
>   Blocked count: 3
>   Waited count: 65
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
>   State: RUNNABLE
>   Blocked count: 43
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
>   State: TIMED_WAITING
>   Blocked count: 1
>   Waited count: 17
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:509)
>     java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>     org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 8
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
>   State: RUNNABLE
>   Blocked count: 45
>   Waited count: 44
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
>   State: TIMED_WAITING
>   Blocked count: 1
>   Waited count: 147
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
>     java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
>   State: TIMED_WAITING
>   Blocked count: 2
>   Waited count: 147
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
>     java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 220
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
>     java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 89
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
>     java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 2
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
>     java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
> Thread 3 (Finalizer):
>   State: WAITING
>   Blocked count: 9
>   Waited count: 10
>   Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
>   State: WAITING
>   Blocked count: 11
>   Waited count: 12
>   Waiting on java.lang.ref.Reference$Lock@270d75a3
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Object.wait(Object.java:485)
>     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
>   State: RUNNABLE
>   Blocked count: 104
>   Waited count: 97
>   Stack:
>     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
>     org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>     org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
>     org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
>     org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
>     org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
>     org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
>     org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
>     sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     java.lang.reflect.Method.invoke(Method.java:597)
>     org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>     org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>     org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>     org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
>     org.junit.runners.ParentRunner.run(ParentRunner.java:236)
>     org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message