hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramkrishna S Vasudevan <ramakrish...@huawei.com>
Subject RE: Possible dead lock
Date Wed, 20 Apr 2011 04:46:52 GMT
Hi

I am using hbase-0.90.1.

The lock i meant was the one present in flushRegion api that gets called as
part of flushOneforGlobalPressure.  Sorry for not being clear.

I am attaching the dump for easy readability.

Regards
Ram



****************************************************************************
***********
This e-mail and attachments contain confidential information from HUAWEI,
which is intended only for the person or entity whose address is listed
above. Any use of the information contained herein in any way (including,
but not limited to, total or partial disclosure, reproduction, or
dissemination) by persons other than the intended recipient's) is
prohibited. If you receive this e-mail in error, please notify the sender by
phone or email immediately and delete it!


-----Original Message-----
From: Ted Yu [mailto:yuzhihong@gmail.com] 
Sent: Wednesday, April 20, 2011 10:03 AM
To: user@hbase.apache.org; ramakrishnas@huawei.com
Subject: Re: Possible dead lock

Which HBase version are you using ?
I don't see lock object in MemStoreFlusher.flushOneForGlobalPressure().

In the future, please make sure stack trace lines aren't wrapped in the
middle.

On Tue, Apr 19, 2011 at 9:16 PM, Ramkrishna S Vasudevan <
ramakrishnas@huawei.com> wrote:

> Possible dead lock in hBase.
>
>
>
> We are using a client to do multiple write operations parallely.
>
>
>
> Here the problem could be like as part of flushforGlobalPressure the lock
> object is obtained and the flow goes into requestCompaction in compact
> split
> thread.
>
> From there there is a an addition to the PriorityCompactionQueue. Here the
> toString of CompactionRequest is getting used.
>
>
>
> Inside this a Data object is used, further digging leads to
ResourceBundle,
> where the currentThread lock is tried to obtain.
>
>
>
> By the time the other other write operation comes and acquires a lock on
> Memstoreflusher thro recliamStoreMemory api.(synchronized).
>
>
>
> Inside the recliamStoreMemory the lock is again tried to obtain but it is
> already obtained as part of flushforGlobalPressure.
>
> So one is waiting for getting the lock object where the one inside the
> ResourceBundle is trying to acquire the current thread which is nothing
but
> the memstoreflusher.
>
>
>
> Pls chk and verify.
>
>
>
> 2011-04-19 14:29:37
>
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):
>
>
>
> "IPC Client (-1912063050) connection to /10.18.52.108:9000 from root"
> daemon
> prio=10 tid=0x00007fe7b8372800 nid=0x64f9 in Object.wait()
> [0x00007fe7be5e4000]
>
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7ca1369b0> (a
> org.apache.hadoop.ipc.Client$Connection)
>
>            at
> org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:545)
>
>            - locked <0x00007fe7ca1369b0> (a
> org.apache.hadoop.ipc.Client$Connection)
>
>            at org.apache.hadoop.ipc.Client$Connection.run(Client.java:593)
>
>
>
> "Attach Listener" daemon prio=10 tid=0x0000000040a01800 nid=0x64ec waiting
> on condition [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "1470390985@qtp-171206900-11" prio=10 tid=0x0000000041201800 nid=0x6466 in
> Object.wait() [0x00007fe7c4316000]
>
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7cbaf6550> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>
>            at
>
>
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626
> )
>
>            - locked <0x00007fe7cbaf6550> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>
>
>
> "RS_OPEN_REGION-linux108,60020,1303201486948-0" daemon prio=10
> tid=0x00000000407e5800 nid=0x63d4 waiting on condition
[0x00007fe7be6e5000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7ccaf1c88> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "ResponseProcessor for block blk_1303201331094_1008" daemon prio=10
> tid=0x00000000407db000 nid=0x63bf runnable [0x00007fe7be7e6000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7ccbbce50> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7ccbbce38> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7ccbbcad8> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at
>
>
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTi
> meout.java:333)
>
>            at
>
>
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)
>
>            at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
>
>            at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
>
>            at java.io.DataInputStream.readFully(DataInputStream.java:178)
>
>            at java.io.DataInputStream.readLong(DataInputStream.java:399)
>
>            at
>
>
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(
> DataTransferProtocol.java:133)
>
>            at
>
>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSCl
> ient.java:3236)
>
>
>
> "LRU Statistics #0" prio=10 tid=0x00007fe7b862a800 nid=0x63bc waiting on
> condition [0x00007fe7be8e7000]
>
>   java.lang.Thread.State: TIMED_WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbac2248> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
>
>            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>
>            at
>
>
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sched
> uledThreadPoolExecutor.java:583)
>
>            at
>
>
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sched
> uledThreadPoolExecutor.java:576)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00007fe7b862a000
> nid=0x63bb in Object.wait() [0x00007fe7be9e8000]
>
>   java.lang.Thread.State: WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7cbb08268> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>
>            at java.lang.Object.wait(Object.java:485)
>
>            at
>
>
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCa
> che.java:519)
>
>            - locked <0x00007fe7cbb08268> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>
>
>
> "PRI IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007fe7b85fd800
> nid=0x63ba waiting on condition [0x00007fe7beae9000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007fe7b85fb800
> nid=0x63b9 waiting on condition [0x00007fe7bebea000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007fe7b85f9800
> nid=0x63b8 waiting on condition [0x00007fe7beceb000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007fe7b85f8000
> nid=0x63b7 waiting on condition [0x00007fe7bedec000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007fe7b85f6000
> nid=0x63b6 waiting on condition [0x00007fe7beeed000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007fe7b85f4000
> nid=0x63b5 waiting on condition [0x00007fe7befee000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007fe7b85f2000
> nid=0x63b4 waiting on condition [0x00007fe7bf0ef000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007fe7b85ef000
> nid=0x63b3 waiting on condition [0x00007fe7bf1f0000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007fe7b85ee800
> nid=0x63b2 waiting on condition [0x00007fe7bf2f1000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "PRI IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007fe7b85ed800
> nid=0x63b1 waiting on condition [0x00007fe7bf3f2000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbad3d10> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)
>
>
>
> "IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007fe7b85ed000
> nid=0x63b0 waiting for monitor entry [0x00007fe7bf4f3000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007fe7b85ec800
> nid=0x63af waiting for monitor entry [0x00007fe7bf5f4000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007fe7b85eb800
> nid=0x63ae waiting on condition [0x00007fe7bf6f5000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbb06228> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(
> AbstractQueuedSynchronizer.java:747)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Abstract
> QueuedSynchronizer.java:778)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueued
> Synchronizer.java:1114)
>
>            at
>
>
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java
> :186)
>
>            at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java:429)
>
>            - locked <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007fe7b85eb000
> nid=0x63ad waiting for monitor entry [0x00007fe7bf7f6000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007fe7b85ea000
> nid=0x63ac waiting for monitor entry [0x00007fe7bf8f7000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007fe7b85e9800
> nid=0x63ab waiting for monitor entry [0x00007fe7bf9f8000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007fe7b854a800
> nid=0x63aa waiting for monitor entry [0x00007fe7bfaf9000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007fe7b854a000
> nid=0x63a9 waiting for monitor entry [0x00007fe7bfbfa000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007fe7b83ba000
> nid=0x63a8 waiting for monitor entry [0x00007fe7bfcfb000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007fe7b8542000
> nid=0x63a7 waiting for monitor entry [0x00007fe7bfdfc000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
>
>
> "IPC Server listener on 60020" daemon prio=10 tid=0x00007fe7b84d2000
> nid=0x63a6 runnable [0x00007fe7bfefd000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbb01528> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbb01510> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf648> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:414)
>
>
>
> "IPC Server Responder" daemon prio=10 tid=0x00007fe7b8549000 nid=0x63a5
> runnable [0x00007fe7bfffe000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbab2d68> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbab2d50> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbaba240> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at
>
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:588)
>
>
>
> "Timer-0" daemon prio=10 tid=0x00007fe7b8544800 nid=0x63a4 in
Object.wait()
> [0x00007fe7c4114000]
>
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7cbacf138> (a java.util.TaskQueue)
>
>            at java.util.TimerThread.mainLoop(Timer.java:509)
>
>            - locked <0x00007fe7cbacf138> (a java.util.TaskQueue)
>
>            at java.util.TimerThread.run(Timer.java:462)
>
>
>
> "251172046@qtp-171206900-1 - Acceptor0
> SelectChannelConnector@0.0.0.0:60030"
> prio=10 tid=0x00007fe7b84e1800 nid=0x63a3 runnable [0x00007fe7c4215000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cc0536c8> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cc0536b0> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7ccb05208> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at
>
>
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:4
> 98)
>
>            at
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
>
>            at
>
>
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.j
> ava:124)
>
>            at
>
>
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
>
>            at
>
>
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582
> )
>
>
>
> "regionserver60020.leaseChecker" daemon prio=10 tid=0x00007fe7b8349000
> nid=0x63a1 waiting on condition [0x00007fe7c4417000]
>
>   java.lang.Thread.State: TIMED_WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbb05168> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
>
>            at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)
>
>            at
> org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:82)
>
>
>
> "regionserver60020.majorCompactionChecker" daemon prio=10
> tid=0x00007fe7b831e000 nid=0x63a0 in Object.wait() [0x00007fe7c4518000]
>
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7ccabcdb0> (a java.lang.Object)
>
>            at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)
>
>            - locked <0x00007fe7ccabcdb0> (a java.lang.Object)
>
>            at org.apache.hadoop.hbase.Chore.run(Chore.java:74)
>
>
>
> "regionserver60020.compactor" daemon prio=10 tid=0x00007fe7b8351000
> nid=0x639f waiting on condition [0x00007fe7c4619000]
>
>   java.lang.Thread.State: TIMED_WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbb051e0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
>
>            at
>
>
java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:2
> 45)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.poll(PriorityCo
> mpactionQueue.java:248)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThre
> ad.java:75)
>
>
>
> "regionserver60020.cacheFlusher" daemon prio=10 tid=0x00007fe7b831b800
> nid=0x639e waiting for monitor entry [0x00007fe7c4719000]
>
>   java.lang.Thread.State: BLOCKED (on object monitor)
>
>            at
java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
>
>            at
java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
>
>            at
> java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
>
>            at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
>
>            at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
>
>            at java.security.AccessController.$$YJP$$doPrivileged(Native
> Method)
>
>            at
> java.security.AccessController.doPrivileged(AccessController.java)
>
>            at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
>
>            at
> sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
>
>            at
> sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
>
>            at
>
>
sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:8
> 0)
>
>            at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
>
>            at java.util.TimeZone.getDisplayName(TimeZone.java:350)
>
>            at java.util.Date.toString(Date.java:1025)
>
>            at java.lang.String.valueOf(String.java:2826)
>
>            at java.lang.StringBuilder.append(StringBuilder.java:115)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionReque
> st.toString(PriorityCompactionQueue.java:114)
>
>            at java.lang.String.valueOf(String.java:2826)
>
>            at java.lang.StringBuilder.append(StringBuilder.java:115)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQ
> ueue(PriorityCompactionQueue.java:145)
>
>            - locked <0x00007fe7ccabd258> (a java.util.HashMap)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCom
> pactionQueue.java:188)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(Co
> mpactSplitThread.java:140)
>
>            - locked <0x00007fe7cbaf08c8> (a
> org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(Co
> mpactSplitThread.java:118)
>
>            - locked <0x00007fe7cbaf08c8> (a
> org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlu
> sher.java:387)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressu
> re(MemStoreFlusher.java:194)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.jav
> a:214)
>
>
>
> "regionserver60020.logRoller" daemon prio=10 tid=0x00007fe7b836d800
> nid=0x639d in Object.wait() [0x00007fe7c481b000]
>
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7cbacdbb0> (a
> java.util.concurrent.atomic.AtomicBoolean)
>
>            at
> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
>
>            - locked <0x00007fe7cbacdbb0> (a
> java.util.concurrent.atomic.AtomicBoolean)
>
>
>
> "regionserver60020.logSyncer" daemon prio=10 tid=0x00007fe7b832e800
> nid=0x639c waiting on condition [0x00007fe7c491c000]
>
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>
>            at java.lang.Thread.$$YJP$$sleep(Native Method)
>
>            at java.lang.Thread.sleep(Thread.java)
>
>            at
> org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:963)
>
>
>
> "LeaseChecker" daemon prio=10 tid=0x00007fe7b831d000 nid=0x639b waiting on
> condition [0x00007fe7c4a1d000]
>
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>
>            at java.lang.Thread.$$YJP$$sleep(Native Method)
>
>            at java.lang.Thread.sleep(Thread.java)
>
>            at
> org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1458)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "DataStreamer for file
> /hbase/.logs/linux108,60020,1303201486948/linux108%3A60020.1303201488470
> block blk_1303201331094_1008" daemon prio=10 tid=0x00007fe7b832c800
> nid=0x639a in Object.wait() [0x00007fe7c4b1e000]
>
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7cbacdf10> (a java.util.LinkedList)
>
>            at
>
>
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.
> java:3036)
>
>            - locked <0x00007fe7cbacdf10> (a java.util.LinkedList)
>
>
>
> "sendParams-0" daemon prio=10 tid=0x00007fe7b832d800 nid=0x6399 waiting on
> condition [0x00007fe7c4c1f000]
>
>   java.lang.Thread.State: TIMED_WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbafc3a8> (a
> java.util.concurrent.SynchronousQueue$TransferStack)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>
>            at
>
>
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchronous
> Queue.java:424)
>
>            at
>
>
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueu
> e.java:323)
>
>            at
> java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 07)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Client (-1912063050) connection to linux253/10.18.52.253:60000 from
> an
> unknown user" daemon prio=10 tid=0x00007fe7b82aa000 nid=0x6394 in
> Object.wait() [0x00007fe7c4d20000]
>
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7cc597be8> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.j
> ava:431)
>
>            - locked <0x00007fe7cc597be8> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:476)
>
>
>
> "DestroyJavaVM" prio=10 tid=0x00007fe7c016b000 nid=0x6361 waiting on
> condition [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "regionserver60020-EventThread" daemon prio=10 tid=0x0000000040520000
> nid=0x6381 waiting on condition [0x00007fe7c4f22000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbb053c0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
>
>
> "regionserver60020-SendThread(10.18.52.96:2191)" daemon prio=10
> tid=0x0000000040425800 nid=0x6380 runnable [0x00007fe7c5023000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7ccabcf30> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7ccabcf18> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7ccac12c8> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>
>
>
> "regionserver60020" prio=10 tid=0x00007fe7b826e800 nid=0x637f waiting on
> condition [0x00007fe7c5124000]
>
>   java.lang.Thread.State: TIMED_WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cc5ba120> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
> anos(AbstractQueuedSynchronizer.java:1963)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:61
> 6)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 9 on port 60020" prio=10 tid=0x00007fe7b8075800 nid=0x637e
> runnable [0x00007fe7c5225000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbab0890> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbab0878> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf198> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbad3bf8> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 8 on port 60020" prio=10 tid=0x00007fe7b8074800 nid=0x637d
> runnable [0x00007fe7c5326000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbadb4f0> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbadb4d8> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf3f0> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbb00d68> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 7 on port 60020" prio=10 tid=0x00007fe7b8074000 nid=0x637c
> runnable [0x00007fe7c5427000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbadc7d0> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbadc7b8> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf5d0> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbb00d40> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 6 on port 60020" prio=10 tid=0x00007fe7b8073800 nid=0x637b
> runnable [0x00007fe7c5528000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbadb610> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbadb5f8> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf468> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbad3f40> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 5 on port 60020" prio=10 tid=0x00007fe7b806f800 nid=0x637a
> runnable [0x00007fe7c5629000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbab0b60> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbab0b48> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf288> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbad3f18> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 4 on port 60020" prio=10 tid=0x00007fe7b8072000 nid=0x6379
> runnable [0x00007fe7c572a000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbab0c80> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbab0c68> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf300> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbad3ef0> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 3 on port 60020" prio=10 tid=0x00007fe7b8083000 nid=0x6378
> runnable [0x00007fe7c582b000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbadc6b0> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbadc698> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf558> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbad3ec8> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 2 on port 60020" prio=10 tid=0x00007fe7b8056800 nid=0x6377
> runnable [0x00007fe7c592c000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbab0a40> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbab0a28> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf210> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbad3ea0> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 1 on port 60020" prio=10 tid=0x00007fe7b806d800 nid=0x6376
> runnable [0x00007fe7c5a2d000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbadb730> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbadb718> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf4e0> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbad3e78> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "IPC Reader 0 on port 60020" prio=10 tid=0x00007fe7b806a800 nid=0x6375
> runnable [0x00007fe7c5b2e000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbadb3d0> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbadb3b8> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf378> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>
>            at
>
>
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
> :305)
>
>            - locked <0x00007fe7cbad3e50> (a
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
> va:886)
>
>            at
>
>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
> 08)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "main-EventThread" daemon prio=10 tid=0x00007fe7b8080800 nid=0x6374
waiting
> on condition [0x00007fe7c6084000]
>
>   java.lang.Thread.State: WAITING (parking)
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbb00e80> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
> AbstractQueuedSynchronizer.java:1925)
>
>            at
>
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>
>            at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
>
>
>
> "main-SendThread(10.18.52.109:2191)" daemon prio=10 tid=0x00007fe7b8063000
> nid=0x6373 runnable [0x00007fe7c6185000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)
>
>            at
> sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)
>
>            at
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>
>            at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>
>            at
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>
>            - locked <0x00007fe7cbb01738> (a sun.nio.ch.Util$1)
>
>            - locked <0x00007fe7cbb01720> (a
> java.util.Collections$UnmodifiableSet)
>
>            - locked <0x00007fe7cbacf6c0> (a sun.nio.ch.EPollSelectorImpl)
>
>            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>
>            at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)
>
>
>
> "Low Memory Detector" daemon prio=10 tid=0x0000000040301000 nid=0x6370
> runnable [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "CompilerThread1" daemon prio=10 tid=0x00000000402e4800 nid=0x636f waiting
> on condition [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "CompilerThread0" daemon prio=10 tid=0x000000004029b800 nid=0x636e waiting
> on condition [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00000000402f4000
> nid=0x636d runnable [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "YJPAgent-CPUSampler" daemon prio=10 tid=0x00000000402eb800 nid=0x636c
> runnable [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "YJPAgent-RequestListener" daemon prio=10 tid=0x00000000402e7800
nid=0x636b
> runnable [0x00007fe7c6b78000]
>
>   java.lang.Thread.State: RUNNABLE
>
>            at java.net.PlainSocketImpl.$$YJP$$socketAccept(Native Method)
>
>            at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)
>
>            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)
>
>            - locked <0x00007fe7cbb01b30> (a java.net.SocksSocketImpl)
>
>            at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>
>            at java.net.ServerSocket.accept(ServerSocket.java:421)
>
>            at com.yourkit.runtime.Core$3.run(Core.java:561)
>
>            at java.lang.Thread.run(Thread.java:619)
>
>
>
> "YJPAgent-Telemetry" daemon prio=10 tid=0x00007fe7c0159800 nid=0x636a
> waiting on condition [0x00007fe7c6ef8000]
>
>   java.lang.Thread.State: TIMED_WAITING (sleeping)
>
>            at java.lang.Thread.$$YJP$$sleep(Native Method)
>
>            at java.lang.Thread.sleep(Thread.java)
>
>            at
>
com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:298)
>
>
>
> "Signal Dispatcher" daemon prio=10 tid=0x00007fe7c0090000 nid=0x6369
> runnable [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00007fe7c00b0000
> nid=0x6368 waiting on condition [0x0000000000000000]
>
>   java.lang.Thread.State: RUNNABLE
>
>
>
> "Finalizer" daemon prio=10 tid=0x00007fe7c0067000 nid=0x6367 in
> Object.wait() [0x00007fe7c71fb000]
>
>   java.lang.Thread.State: WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7cc00ad28> (a
> java.lang.ref.ReferenceQueue$Lock)
>
>            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>
>            - locked <0x00007fe7cc00ad28> (a
> java.lang.ref.ReferenceQueue$Lock)
>
>            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>
>            at
> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
>
>
> "Reference Handler" daemon prio=10 tid=0x00007fe7c0040000 nid=0x6366 in
> Object.wait() [0x00007fe7c72fc000]
>
>   java.lang.Thread.State: WAITING (on object monitor)
>
>            at java.lang.Object.wait(Native Method)
>
>            - waiting on <0x00007fe7cc00ad58> (a
> java.lang.ref.Reference$Lock)
>
>            at java.lang.Object.wait(Object.java:485)
>
>            at
> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>
>            - locked <0x00007fe7cc00ad58> (a java.lang.ref.Reference$Lock)
>
>
>
> "VM Thread" prio=10 tid=0x00007fe7c0050000 nid=0x6365 runnable
>
>
>
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040173000
> nid=0x6362 runnable
>
>
>
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000040175000
> nid=0x6363 runnable
>
>
>
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x0000000040248000
nid=0x6364
> runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x00007fe7c00f3000 nid=0x6371
waiting
> on condition
>
>
>
> JNI global references: 1304
>
>
>
>
>
> Found one Java-level deadlock:
>
> =============================
>
> "IPC Server handler 9 on 60020":
>
>  waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>
>  which is held by "IPC Server handler 7 on 60020"
>
> "IPC Server handler 7 on 60020":
>
>  waiting for ownable synchronizer 0x00007fe7cbb06228, (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync),
>
>  which is held by "regionserver60020.cacheFlusher"
>
> "regionserver60020.cacheFlusher":
>
>  waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher),
>
>  which is held by "IPC Server handler 7 on 60020"
>
>
>
> Java stack information for the threads listed above:
>
> ===================================================
>
> "IPC Server handler 9 on 60020":
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
> "IPC Server handler 7 on 60020":
>
>            at sun.misc.Unsafe.$$YJP$$park(Native Method)
>
>            - parking to wait for  <0x00007fe7cbb06228> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
>            at sun.misc.Unsafe.park(Unsafe.java)
>
>            at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(
> AbstractQueuedSynchronizer.java:747)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Abstract
> QueuedSynchronizer.java:778)
>
>            at
>
>
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueued
> Synchronizer.java:1114)
>
>            at
>
>
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java
> :186)
>
>            at
> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
> emStoreFlusher.java:429)
>
>            - locked <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2558)
>
>            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
>
>            at
>
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>
>            at java.lang.reflect.Method.invoke(Method.java:597)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>
>            at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>
> "regionserver60020.cacheFlusher":
>
>            at
java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)
>
>            - waiting to lock <0x00007fe7cbacbd48> (a
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher)
>
>            at
java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)
>
>            at
java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)
>
>            at
> java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)
>
>            at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)
>
>            at sun.util.resources.LocaleData$1.run(LocaleData.java:127)
>
>            at java.security.AccessController.$$YJP$$doPrivileged(Native
> Method)
>
>            at
> java.security.AccessController.doPrivileged(AccessController.java)
>
>            at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)
>
>            at
> sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)
>
>            at
> sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)
>
>            at
>
>
sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:8
> 0)
>
>            at java.util.TimeZone.getDisplayNames(TimeZone.java:399)
>
>            at java.util.TimeZone.getDisplayName(TimeZone.java:350)
>
>            at java.util.Date.toString(Date.java:1025)
>
>            at java.lang.String.valueOf(String.java:2826)
>
>            at java.lang.StringBuilder.append(StringBuilder.java:115)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionReque
> st.toString(PriorityCompactionQueue.java:114)
>
>            at java.lang.String.valueOf(String.java:2826)
>
>            at java.lang.StringBuilder.append(StringBuilder.java:115)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQ
> ueue(PriorityCompactionQueue.java:145)
>
>            - locked <0x00007fe7ccabd258> (a java.util.HashMap)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCom
> pactionQueue.java:188)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(Co
> mpactSplitThread.java:140)
>
>            - locked <0x00007fe7cbaf08c8> (a
> org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(Co
> mpactSplitThread.java:118)
>
>            - locked <0x00007fe7cbaf08c8> (a
> org.apache.hadoop.hbase.regionserver.CompactSplitThread)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlu
> sher.java:387)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressu
> re(MemStoreFlusher.java:194)
>
>            at
>
>
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.jav
> a:214)
>
>
>
> Found 1 deadlock.
>
>
>
>

Mime
View raw message