hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nkeywal (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-4724) TestAdmin hangs randomly in trunk
Date Fri, 10 Feb 2012 14:57:04 GMT

     [ https://issues.apache.org/jira/browse/HBASE-4724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

nkeywal updated HBASE-4724:
---------------------------

    Resolution: Fixed
        Status: Resolved  (was: Patch Available)
    
> TestAdmin hangs randomly in trunk
> ---------------------------------
>
>                 Key: HBASE-4724
>                 URL: https://issues.apache.org/jira/browse/HBASE-4724
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 0.94.0
>            Reporter: nkeywal
>            Assignee: nkeywal
>         Attachments: 20111102_4724_TestAdmin.patch, 20111102_4724_TestAdmin.v2.patch
>
>
> fom the logs in my env
> {noformat}
> 2011-11-01 15:48:40,744 WARN  [Master:0;localhost,39664,1320187706355] master.AssignmentManager(1471): Failed assignment of -ROOT-,,0.70236052 to localhost,44046,1320187706849, trying to assign elsewhere instead; retry=1
> org.apache.hadoop.hbase.ipc.HBaseRPC$VersionMismatch: Protocol org.apache.hadoop.hbase.ipc.HRegionInterface version mismatch. (client = 28, server = 29)
> 	at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:185)
> 	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:300)
> {noformat}
> Anyway, after this the logs finishes with:
> {noformat}
> 2011-11-01 15:54:35,132 INFO  [Master:0;localhost,39664,1320187706355.oldLogCleaner] hbase.Chore(80): Master:0;localhost,39664,1320187706355.oldLogCleaner exiting
> Process Thread Dump: Automatic Stack Trace every 60 seconds waiting on Master:0;localhost,39664,1320187706355
> {noformat}
> it's in
> {noformat}
>     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
>     org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>     org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
>     org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
>     org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
>     org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
>     org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
>     org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
> {noformat}
> So that's at least why adding a timeout wont help and may be why it does not end at all. Adding a maximum retry to Threads#threadDumpingIsAlive could help.
> I also wonder if the root cause of the non ending is my modif on the wal, with some threads surprised to have updates that were not written in the wal. Here is the full stack dump:
> {noformat}
> Thread 354 (IPC Client (47) connection to localhost/127.0.0.1:52227 from nkeywal):
>   State: TIMED_WAITING
>   Blocked count: 360
>   Waited count: 359
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:702)
>     org.apache.hadoop.ipc.Client$Connection.run(Client.java:744)
> Thread 272 (Master:0;localhost,39664,1320187706355-EventThread):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 4
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@107b954b
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 271 (Master:0;localhost,39664,1320187706355-SendThread(localhost:21819)):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 152 (Master:0;localhost,39664,1320187706355):
>   State: WAITING
>   Blocked count: 217
>   Waited count: 174
>   Waiting on org.apache.hadoop.hbase.zookeeper.RootRegionTracker@6621477c
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Object.wait(Object.java:485)
>     org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:131)
>     org.apache.hadoop.hbase.zookeeper.ZooKeeperNodeTracker.blockUntilAvailable(ZooKeeperNodeTracker.java:104)
>     org.apache.hadoop.hbase.catalog.CatalogTracker.waitForRoot(CatalogTracker.java:277)
>     org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:523)
>     org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:468)
>     org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:309)
>     java.lang.Thread.run(Thread.java:662)
> Thread 165 (LruBlockCache.EvictionThread):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@3e9d7b56
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Object.wait(Object.java:485)
>     org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:593)
>     java.lang.Thread.run(Thread.java:662)
> Thread 151 (main-EventThread):
>   State: WAITING
>   Blocked count: 40
>   Waited count: 39
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22531380
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
> Thread 150 (main-SendThread(localhost:21819)):
>   State: RUNNABLE
>   Blocked count: 55
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
> Thread 149 (IPC Server handler 4 on 39664):
>   State: WAITING
>   Blocked count: 3
>   Waited count: 1968
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 148 (IPC Server handler 3 on 39664):
>   State: WAITING
>   Blocked count: 7
>   Waited count: 1969
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 147 (IPC Server handler 2 on 39664):
>   State: WAITING
>   Blocked count: 11
>   Waited count: 1968
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 146 (IPC Server handler 1 on 39664):
>   State: WAITING
>   Blocked count: 6
>   Waited count: 1969
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 145 (IPC Server handler 0 on 39664):
>   State: WAITING
>   Blocked count: 3
>   Waited count: 1970
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73f5173f
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1224)
> Thread 133 (IPC Server listener on 39664):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:580)
> Thread 144 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:754)
> Thread 143 (IPC Reader 9 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 142 (IPC Reader 8 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 141 (IPC Reader 7 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 140 (IPC Reader 6 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 139 (IPC Reader 5 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 138 (IPC Reader 4 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 137 (IPC Reader 3 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 136 (IPC Reader 2 on port 39664):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 135 (IPC Reader 1 on port 39664):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 134 (IPC Reader 0 on port 39664):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:471)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 124 (LeaseChecker):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 407
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1252)
>     java.lang.Thread.run(Thread.java:662)
> Thread 121 (ProcessThread:-1):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 269
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f5d84e0
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:103)
> Thread 120 (SyncThread:0):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 252
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b25b27c
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:94)
> Thread 119 (SessionTracker):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 207
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:142)
> Thread 118 (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:21819):
>   State: RUNNABLE
>   Blocked count: 38
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:232)
> Thread 116 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@2e6f947b):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 407
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
>     java.lang.Thread.run(Thread.java:662)
> Thread 113 (IPC Server handler 2 on 39899):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 112 (IPC Server handler 1 on 39899):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 111 (IPC Server handler 0 on 39899):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b4c72c8
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 106 (IPC Server listener on 39899):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 108 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 102 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@45d1c3cd):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>     sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
>     sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>     org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
>     java.lang.Thread.run(Thread.java:662)
> Thread 109 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6]):
>   State: TIMED_WAITING
>   Blocked count: 137
>   Waited count: 283
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
>     java.lang.Thread.run(Thread.java:662)
> Thread 107 (pool-4-thread-1):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 105 (Timer-3):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 15
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:509)
>     java.util.TimerThread.run(Timer.java:462)
> Thread 104 (1353906974@qtp-1197660496-1 - Acceptor0 SelectChannelConnector@localhost:59858):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>     org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 103 (1185716172@qtp-1197660496-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 8
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 101 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data6):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 98 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data5):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 90 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@605b28c9):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 408
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
>     java.lang.Thread.run(Thread.java:662)
> Thread 86 (IPC Server handler 2 on 56098):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 85 (IPC Server handler 1 on 56098):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 84 (IPC Server handler 0 on 56098):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@78ff22ed
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 80 (IPC Server listener on 56098):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 82 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 76 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@2b68989e):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>     sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
>     sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>     org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
>     java.lang.Thread.run(Thread.java:662)
> Thread 83 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4]):
>   State: TIMED_WAITING
>   Blocked count: 149
>   Waited count: 290
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
>     java.lang.Thread.run(Thread.java:662)
> Thread 81 (pool-3-thread-1):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 79 (Timer-2):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 15
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:509)
>     java.util.TimerThread.run(Timer.java:462)
> Thread 78 (348878159@qtp-164967086-1 - Acceptor0 SelectChannelConnector@localhost:36427):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>     org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 77 (1905729203@qtp-164967086-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 8
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 75 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data4):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 72 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data3):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 61 (org.apache.hadoop.hdfs.server.datanode.DataBlockScanner@6709da93):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 408
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:620)
>     java.lang.Thread.run(Thread.java:662)
> Thread 60 (IPC Server handler 2 on 40388):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 59 (IPC Server handler 1 on 40388):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 58 (IPC Server handler 0 on 40388):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 1
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56afd9e3
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 53 (IPC Server listener on 40388):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 55 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 49 (org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@48baa31b):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
>     sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
>     sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84)
>     org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131)
>     java.lang.Thread.run(Thread.java:662)
> Thread 57 (DataNode: [/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1,/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2]):
>   State: TIMED_WAITING
>   Blocked count: 148
>   Waited count: 283
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.offerService(DataNode.java:986)
>     org.apache.hadoop.hdfs.server.datanode.DataNode.run(DataNode.java:1421)
>     java.lang.Thread.run(Thread.java:662)
> Thread 54 (pool-2-thread-1):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 52 (Timer-1):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 15
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:509)
>     java.util.TimerThread.run(Timer.java:462)
> Thread 51 (1031841975@qtp-1068922666-1 - Acceptor0 SelectChannelConnector@localhost:56535):
>   State: RUNNABLE
>   Blocked count: 1
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>     org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 50 (1191055321@qtp-1068922666-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 8
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 48 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data2):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 45 (refreshUsed-/tmp/d846d12d-19f3-439a-8cab-e05db67ae60b/dfscluster_2773516a-bc5a-4029-9ee0-fc499eee6ca8/dfs/data/data1):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 1
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.fs.DU$DURefreshThread.run(DU.java:80)
>     java.lang.Thread.run(Thread.java:662)
> Thread 31 (IPC Server handler 9 on 52227):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 64
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 30 (IPC Server handler 8 on 52227):
>   State: WAITING
>   Blocked count: 3
>   Waited count: 65
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 29 (IPC Server handler 7 on 52227):
>   State: WAITING
>   Blocked count: 2
>   Waited count: 66
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 28 (IPC Server handler 6 on 52227):
>   State: WAITING
>   Blocked count: 1
>   Waited count: 64
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 27 (IPC Server handler 5 on 52227):
>   State: WAITING
>   Blocked count: 4
>   Waited count: 68
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 26 (IPC Server handler 4 on 52227):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 64
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 25 (IPC Server handler 3 on 52227):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 65
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 24 (IPC Server handler 2 on 52227):
>   State: WAITING
>   Blocked count: 1
>   Waited count: 66
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 23 (IPC Server handler 1 on 52227):
>   State: WAITING
>   Blocked count: 0
>   Waited count: 64
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 22 (IPC Server handler 0 on 52227):
>   State: WAITING
>   Blocked count: 3
>   Waited count: 65
>   Waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@64473a14
>   Stack:
>     sun.misc.Unsafe.park(Native Method)
>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>     java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
>     java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
>     org.apache.hadoop.ipc.Server$Handler.run(Server.java:1364)
> Thread 15 (IPC Server listener on 52227):
>   State: RUNNABLE
>   Blocked count: 43
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener.run(Server.java:439)
> Thread 17 (IPC Server Responder):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.apache.hadoop.ipc.Server$Responder.run(Server.java:605)
> Thread 21 (Timer-0):
>   State: TIMED_WAITING
>   Blocked count: 1
>   Waited count: 17
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.util.TimerThread.mainLoop(Timer.java:509)
>     java.util.TimerThread.run(Timer.java:462)
> Thread 20 (245022965@qtp-589734556-1 - Acceptor0 SelectChannelConnector@localhost:33079):
>   State: RUNNABLE
>   Blocked count: 2
>   Waited count: 1
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
>     org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>     org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>     org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Thread 19 (1821050251@qtp-589734556-0):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 8
>   Stack:
>     java.lang.Object.wait(Native Method)
>     org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626)
> Thread 16 (pool-1-thread-1):
>   State: RUNNABLE
>   Blocked count: 45
>   Waited count: 44
>   Stack:
>     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210)
>     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>     org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:333)
>     java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     java.lang.Thread.run(Thread.java:662)
> Thread 14 (org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor@620fa83):
>   State: TIMED_WAITING
>   Blocked count: 1
>   Waited count: 147
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
>     java.lang.Thread.run(Thread.java:662)
> Thread 13 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@d5d4de6):
>   State: TIMED_WAITING
>   Blocked count: 2
>   Waited count: 147
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor.run(FSNamesystem.java:2689)
>     java.lang.Thread.run(Thread.java:662)
> Thread 12 (org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor@73d32e45):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 220
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager.java:368)
>     java.lang.Thread.run(Thread.java:662)
> Thread 11 (org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor@19be4777):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 89
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.FSNamesystem$HeartbeatMonitor.run(FSNamesystem.java:2666)
>     java.lang.Thread.run(Thread.java:662)
> Thread 10 (org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor@658f7386):
>   State: TIMED_WAITING
>   Blocked count: 0
>   Waited count: 2
>   Stack:
>     java.lang.Thread.sleep(Native Method)
>     org.apache.hadoop.hdfs.server.namenode.PendingReplicationBlocks$PendingReplicationMonitor.run(PendingReplicationBlocks.java:186)
>     java.lang.Thread.run(Thread.java:662)
> Thread 4 (Signal Dispatcher):
>   State: RUNNABLE
>   Blocked count: 0
>   Waited count: 0
>   Stack:
> Thread 3 (Finalizer):
>   State: WAITING
>   Blocked count: 9
>   Waited count: 10
>   Waiting on java.lang.ref.ReferenceQueue$Lock@3d1a70a7
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>     java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>     java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
> Thread 2 (Reference Handler):
>   State: WAITING
>   Blocked count: 11
>   Waited count: 12
>   Waiting on java.lang.ref.Reference$Lock@270d75a3
>   Stack:
>     java.lang.Object.wait(Native Method)
>     java.lang.Object.wait(Object.java:485)
>     java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> Thread 1 (main):
>   State: RUNNABLE
>   Blocked count: 104
>   Waited count: 97
>   Stack:
>     sun.management.ThreadImpl.getThreadInfo1(Native Method)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:156)
>     sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:121)
>     org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
>     org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:113)
>     org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:405)
>     org.apache.hadoop.hbase.MiniHBaseCluster.join(MiniHBaseCluster.java:408)
>     org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:616)
>     org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:590)
>     org.apache.hadoop.hbase.client.TestAdmin.tearDownAfterClass(TestAdmin.java:89)
>     sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     java.lang.reflect.Method.invoke(Method.java:597)
>     org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>     org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>     org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>     org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
>     org.junit.runners.ParentRunner.run(ParentRunner.java:236)
>     org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53)
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message