hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramkrishna S Vasudevan <ramakrish...@huawei.com>
Subject Possible dead lock
Date Wed, 20 Apr 2011 04:16:21 GMT
Possible dead lock in hBase.

 

We are using a client to do multiple write operations parallely.

 

Here the problem could be like as part of flushforGlobalPressure the lock
object is obtained and the flow goes into requestCompaction in compact split
thread.

>From there there is a an addition to the PriorityCompactionQueue. Here the
toString of CompactionRequest is getting used.

 

Inside this a Data object is used, further digging leads to ResourceBundle,
where the currentThread lock is tried to obtain.

 

By the time the other other write operation comes and acquires a lock on
Memstoreflusher thro recliamStoreMemory api.(synchronized).

 

Inside the recliamStoreMemory the lock is again tried to obtain but it is
already obtained as part of flushforGlobalPressure.

So one is waiting for getting the lock object where the one inside the
ResourceBundle is trying to acquire the current thread which is nothing but
the memstoreflusher.

 

Pls chk and verify.  

 

2011-04-19 14:29:37

Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.3-b01 mixed mode):

 

"IPC Client (-1912063050) connection to /10.18.52.108:9000 from root" daemon
prio=10 tid=0x00007fe7b8372800 nid=0x64f9 in Object.wait()
[0x00007fe7be5e4000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7ca1369b0> (a
org.apache.hadoop.ipc.Client$Connection)

            at
org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:545)

            - locked <0x00007fe7ca1369b0> (a
org.apache.hadoop.ipc.Client$Connection)

            at org.apache.hadoop.ipc.Client$Connection.run(Client.java:593)

 

"Attach Listener" daemon prio=10 tid=0x0000000040a01800 nid=0x64ec waiting
on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"1470390985@qtp-171206900-11" prio=10 tid=0x0000000041201800 nid=0x6466 in
Object.wait() [0x00007fe7c4316000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7cbaf6550> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)

            at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:626
)

            - locked <0x00007fe7cbaf6550> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)

 

"RS_OPEN_REGION-linux108,60020,1303201486948-0" daemon prio=10
tid=0x00000000407e5800 nid=0x63d4 waiting on condition [0x00007fe7be6e5000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7ccaf1c88> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)

            at java.lang.Thread.run(Thread.java:619)

 

"ResponseProcessor for block blk_1303201331094_1008" daemon prio=10
tid=0x00000000407db000 nid=0x63bf runnable [0x00007fe7be7e6000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7ccbbce50> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7ccbbce38> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7ccbbcad8> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at
org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTi
meout.java:333)

            at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:158)

            at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)

            at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)

            at java.io.DataInputStream.readFully(DataInputStream.java:178)

            at java.io.DataInputStream.readLong(DataInputStream.java:399)

            at
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.readFields(
DataTransferProtocol.java:133)

            at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSCl
ient.java:3236)

 

"LRU Statistics #0" prio=10 tid=0x00007fe7b862a800 nid=0x63bc waiting on
condition [0x00007fe7be8e7000]

   java.lang.Thread.State: TIMED_WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbac2248> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)

            at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)

            at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sched
uledThreadPoolExecutor.java:583)

            at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(Sched
uledThreadPoolExecutor.java:576)

            at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)

            at java.lang.Thread.run(Thread.java:619)

 

"LruBlockCache.EvictionThread" daemon prio=10 tid=0x00007fe7b862a000
nid=0x63bb in Object.wait() [0x00007fe7be9e8000]

   java.lang.Thread.State: WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7cbb08268> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)

            at java.lang.Object.wait(Object.java:485)

            at
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCa
che.java:519)

            - locked <0x00007fe7cbb08268> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)

 

"PRI IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007fe7b85fd800
nid=0x63ba waiting on condition [0x00007fe7beae9000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007fe7b85fb800
nid=0x63b9 waiting on condition [0x00007fe7bebea000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007fe7b85f9800
nid=0x63b8 waiting on condition [0x00007fe7beceb000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007fe7b85f8000
nid=0x63b7 waiting on condition [0x00007fe7bedec000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007fe7b85f6000
nid=0x63b6 waiting on condition [0x00007fe7beeed000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007fe7b85f4000
nid=0x63b5 waiting on condition [0x00007fe7befee000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007fe7b85f2000
nid=0x63b4 waiting on condition [0x00007fe7bf0ef000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007fe7b85ef000
nid=0x63b3 waiting on condition [0x00007fe7bf1f0000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007fe7b85ee800
nid=0x63b2 waiting on condition [0x00007fe7bf2f1000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"PRI IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007fe7b85ed800
nid=0x63b1 waiting on condition [0x00007fe7bf3f2000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbad3d10> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1025)

 

"IPC Server handler 9 on 60020" daemon prio=10 tid=0x00007fe7b85ed000
nid=0x63b0 waiting for monitor entry [0x00007fe7bf4f3000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 8 on 60020" daemon prio=10 tid=0x00007fe7b85ec800
nid=0x63af waiting for monitor entry [0x00007fe7bf5f4000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 7 on 60020" daemon prio=10 tid=0x00007fe7b85eb800
nid=0x63ae waiting on condition [0x00007fe7bf6f5000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbb06228> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(
AbstractQueuedSynchronizer.java:747)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Abstract
QueuedSynchronizer.java:778)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueued
Synchronizer.java:1114)

            at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java
:186)

            at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java:429)

            - locked <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 6 on 60020" daemon prio=10 tid=0x00007fe7b85eb000
nid=0x63ad waiting for monitor entry [0x00007fe7bf7f6000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 5 on 60020" daemon prio=10 tid=0x00007fe7b85ea000
nid=0x63ac waiting for monitor entry [0x00007fe7bf8f7000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 4 on 60020" daemon prio=10 tid=0x00007fe7b85e9800
nid=0x63ab waiting for monitor entry [0x00007fe7bf9f8000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 3 on 60020" daemon prio=10 tid=0x00007fe7b854a800
nid=0x63aa waiting for monitor entry [0x00007fe7bfaf9000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 2 on 60020" daemon prio=10 tid=0x00007fe7b854a000
nid=0x63a9 waiting for monitor entry [0x00007fe7bfbfa000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 1 on 60020" daemon prio=10 tid=0x00007fe7b83ba000
nid=0x63a8 waiting for monitor entry [0x00007fe7bfcfb000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server handler 0 on 60020" daemon prio=10 tid=0x00007fe7b8542000
nid=0x63a7 waiting for monitor entry [0x00007fe7bfdfc000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

 

"IPC Server listener on 60020" daemon prio=10 tid=0x00007fe7b84d2000
nid=0x63a6 runnable [0x00007fe7bfefd000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbb01528> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbb01510> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf648> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:414)

 

"IPC Server Responder" daemon prio=10 tid=0x00007fe7b8549000 nid=0x63a5
runnable [0x00007fe7bfffe000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbab2d68> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbab2d50> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbaba240> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:588)

 

"Timer-0" daemon prio=10 tid=0x00007fe7b8544800 nid=0x63a4 in Object.wait()
[0x00007fe7c4114000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7cbacf138> (a java.util.TaskQueue)

            at java.util.TimerThread.mainLoop(Timer.java:509)

            - locked <0x00007fe7cbacf138> (a java.util.TaskQueue)

            at java.util.TimerThread.run(Timer.java:462)

 

"251172046@qtp-171206900-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60030"
prio=10 tid=0x00007fe7b84e1800 nid=0x63a3 runnable [0x00007fe7c4215000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cc0536c8> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cc0536b0> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7ccb05208> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:4
98)

            at
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)

            at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.j
ava:124)

            at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)

            at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582
)

 

"regionserver60020.leaseChecker" daemon prio=10 tid=0x00007fe7b8349000
nid=0x63a1 waiting on condition [0x00007fe7c4417000]

   java.lang.Thread.State: TIMED_WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbb05168> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)

            at java.util.concurrent.DelayQueue.poll(DelayQueue.java:201)

            at
org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:82)

 

"regionserver60020.majorCompactionChecker" daemon prio=10
tid=0x00007fe7b831e000 nid=0x63a0 in Object.wait() [0x00007fe7c4518000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7ccabcdb0> (a java.lang.Object)

            at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:91)

            - locked <0x00007fe7ccabcdb0> (a java.lang.Object)

            at org.apache.hadoop.hbase.Chore.run(Chore.java:74)

 

"regionserver60020.compactor" daemon prio=10 tid=0x00007fe7b8351000
nid=0x639f waiting on condition [0x00007fe7c4619000]

   java.lang.Thread.State: TIMED_WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbb051e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)

            at
java.util.concurrent.PriorityBlockingQueue.poll(PriorityBlockingQueue.java:2
45)

            at
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.poll(PriorityCo
mpactionQueue.java:248)

            at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThre
ad.java:75)

 

"regionserver60020.cacheFlusher" daemon prio=10 tid=0x00007fe7b831b800
nid=0x639e waiting for monitor entry [0x00007fe7c4719000]

   java.lang.Thread.State: BLOCKED (on object monitor)

            at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)

            at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)

            at
java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)

            at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)

            at sun.util.resources.LocaleData$1.run(LocaleData.java:127)

            at java.security.AccessController.$$YJP$$doPrivileged(Native
Method)

            at
java.security.AccessController.doPrivileged(AccessController.java)

            at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)

            at
sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)

            at
sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)

            at
sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:8
0)

            at java.util.TimeZone.getDisplayNames(TimeZone.java:399)

            at java.util.TimeZone.getDisplayName(TimeZone.java:350)

            at java.util.Date.toString(Date.java:1025)

            at java.lang.String.valueOf(String.java:2826)

            at java.lang.StringBuilder.append(StringBuilder.java:115)

            at
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionReque
st.toString(PriorityCompactionQueue.java:114)

            at java.lang.String.valueOf(String.java:2826)

            at java.lang.StringBuilder.append(StringBuilder.java:115)

            at
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQ
ueue(PriorityCompactionQueue.java:145)

            - locked <0x00007fe7ccabd258> (a java.util.HashMap)

            at
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCom
pactionQueue.java:188)

            at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(Co
mpactSplitThread.java:140)

            - locked <0x00007fe7cbaf08c8> (a
org.apache.hadoop.hbase.regionserver.CompactSplitThread)

            at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(Co
mpactSplitThread.java:118)

            - locked <0x00007fe7cbaf08c8> (a
org.apache.hadoop.hbase.regionserver.CompactSplitThread)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlu
sher.java:387)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressu
re(MemStoreFlusher.java:194)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.jav
a:214)

 

"regionserver60020.logRoller" daemon prio=10 tid=0x00007fe7b836d800
nid=0x639d in Object.wait() [0x00007fe7c481b000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7cbacdbb0> (a
java.util.concurrent.atomic.AtomicBoolean)

            at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)

            - locked <0x00007fe7cbacdbb0> (a
java.util.concurrent.atomic.AtomicBoolean)

 

"regionserver60020.logSyncer" daemon prio=10 tid=0x00007fe7b832e800
nid=0x639c waiting on condition [0x00007fe7c491c000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

            at java.lang.Thread.$$YJP$$sleep(Native Method)

            at java.lang.Thread.sleep(Thread.java)

            at
org.apache.hadoop.hbase.regionserver.wal.HLog$LogSyncer.run(HLog.java:963)

 

"LeaseChecker" daemon prio=10 tid=0x00007fe7b831d000 nid=0x639b waiting on
condition [0x00007fe7c4a1d000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

            at java.lang.Thread.$$YJP$$sleep(Native Method)

            at java.lang.Thread.sleep(Thread.java)

            at
org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1458)

            at java.lang.Thread.run(Thread.java:619)

 

"DataStreamer for file
/hbase/.logs/linux108,60020,1303201486948/linux108%3A60020.1303201488470
block blk_1303201331094_1008" daemon prio=10 tid=0x00007fe7b832c800
nid=0x639a in Object.wait() [0x00007fe7c4b1e000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7cbacdf10> (a java.util.LinkedList)

            at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.
java:3036)

            - locked <0x00007fe7cbacdf10> (a java.util.LinkedList)

 

"sendParams-0" daemon prio=10 tid=0x00007fe7b832d800 nid=0x6399 waiting on
condition [0x00007fe7c4c1f000]

   java.lang.Thread.State: TIMED_WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbafc3a8> (a
java.util.concurrent.SynchronousQueue$TransferStack)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

            at
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(Synchronous
Queue.java:424)

            at
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueu
e.java:323)

            at
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:874)

            at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
07)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Client (-1912063050) connection to linux253/10.18.52.253:60000 from an
unknown user" daemon prio=10 tid=0x00007fe7b82aa000 nid=0x6394 in
Object.wait() [0x00007fe7c4d20000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7cc597be8> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Connection)

            at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.j
ava:431)

            - locked <0x00007fe7cc597be8> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Connection)

            at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:476)

 

"DestroyJavaVM" prio=10 tid=0x00007fe7c016b000 nid=0x6361 waiting on
condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"regionserver60020-EventThread" daemon prio=10 tid=0x0000000040520000
nid=0x6381 waiting on condition [0x00007fe7c4f22000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbb053c0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

 

"regionserver60020-SendThread(10.18.52.96:2191)" daemon prio=10
tid=0x0000000040425800 nid=0x6380 runnable [0x00007fe7c5023000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7ccabcf30> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7ccabcf18> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7ccac12c8> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)

 

"regionserver60020" prio=10 tid=0x00007fe7b826e800 nid=0x637f waiting on
condition [0x00007fe7c5124000]

   java.lang.Thread.State: TIMED_WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cc5ba120> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitN
anos(AbstractQueuedSynchronizer.java:1963)

            at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:61
6)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 9 on port 60020" prio=10 tid=0x00007fe7b8075800 nid=0x637e
runnable [0x00007fe7c5225000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbab0890> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbab0878> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf198> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbad3bf8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 8 on port 60020" prio=10 tid=0x00007fe7b8074800 nid=0x637d
runnable [0x00007fe7c5326000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbadb4f0> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbadb4d8> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf3f0> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbb00d68> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 7 on port 60020" prio=10 tid=0x00007fe7b8074000 nid=0x637c
runnable [0x00007fe7c5427000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbadc7d0> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbadc7b8> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf5d0> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbb00d40> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 6 on port 60020" prio=10 tid=0x00007fe7b8073800 nid=0x637b
runnable [0x00007fe7c5528000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbadb610> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbadb5f8> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf468> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbad3f40> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 5 on port 60020" prio=10 tid=0x00007fe7b806f800 nid=0x637a
runnable [0x00007fe7c5629000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbab0b60> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbab0b48> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf288> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbad3f18> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 4 on port 60020" prio=10 tid=0x00007fe7b8072000 nid=0x6379
runnable [0x00007fe7c572a000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbab0c80> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbab0c68> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf300> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbad3ef0> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 3 on port 60020" prio=10 tid=0x00007fe7b8083000 nid=0x6378
runnable [0x00007fe7c582b000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbadc6b0> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbadc698> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf558> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbad3ec8> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 2 on port 60020" prio=10 tid=0x00007fe7b8056800 nid=0x6377
runnable [0x00007fe7c592c000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbab0a40> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbab0a28> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf210> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbad3ea0> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 1 on port 60020" prio=10 tid=0x00007fe7b806d800 nid=0x6376
runnable [0x00007fe7c5a2d000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbadb730> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbadb718> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf4e0> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbad3e78> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"IPC Reader 0 on port 60020" prio=10 tid=0x00007fe7b806a800 nid=0x6375
runnable [0x00007fe7c5b2e000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbadb3d0> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbadb3b8> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf378> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java
:305)

            - locked <0x00007fe7cbad3e50> (a
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja
va:886)

            at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9
08)

            at java.lang.Thread.run(Thread.java:619)

 

"main-EventThread" daemon prio=10 tid=0x00007fe7b8080800 nid=0x6374 waiting
on condition [0x00007fe7c6084000]

   java.lang.Thread.State: WAITING (parking)

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbb00e80> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:1925)

            at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)

            at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)

 

"main-SendThread(10.18.52.109:2191)" daemon prio=10 tid=0x00007fe7b8063000
nid=0x6373 runnable [0x00007fe7c6185000]

   java.lang.Thread.State: RUNNABLE

            at sun.nio.ch.EPollArrayWrapper.$$YJP$$epollWait(Native Method)

            at
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java)

            at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)

            at
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)

            at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

            - locked <0x00007fe7cbb01738> (a sun.nio.ch.Util$1)

            - locked <0x00007fe7cbb01720> (a
java.util.Collections$UnmodifiableSet)

            - locked <0x00007fe7cbacf6c0> (a sun.nio.ch.EPollSelectorImpl)

            at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)

            at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1107)

 

"Low Memory Detector" daemon prio=10 tid=0x0000000040301000 nid=0x6370
runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"CompilerThread1" daemon prio=10 tid=0x00000000402e4800 nid=0x636f waiting
on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"CompilerThread0" daemon prio=10 tid=0x000000004029b800 nid=0x636e waiting
on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"YJPAgent-OOMESnapshotDetector" daemon prio=10 tid=0x00000000402f4000
nid=0x636d runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"YJPAgent-CPUSampler" daemon prio=10 tid=0x00000000402eb800 nid=0x636c
runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"YJPAgent-RequestListener" daemon prio=10 tid=0x00000000402e7800 nid=0x636b
runnable [0x00007fe7c6b78000]

   java.lang.Thread.State: RUNNABLE

            at java.net.PlainSocketImpl.$$YJP$$socketAccept(Native Method)

            at java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java)

            at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390)

            - locked <0x00007fe7cbb01b30> (a java.net.SocksSocketImpl)

            at java.net.ServerSocket.implAccept(ServerSocket.java:453)

            at java.net.ServerSocket.accept(ServerSocket.java:421)

            at com.yourkit.runtime.Core$3.run(Core.java:561)

            at java.lang.Thread.run(Thread.java:619)

 

"YJPAgent-Telemetry" daemon prio=10 tid=0x00007fe7c0159800 nid=0x636a
waiting on condition [0x00007fe7c6ef8000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

            at java.lang.Thread.$$YJP$$sleep(Native Method)

            at java.lang.Thread.sleep(Thread.java)

            at
com.yourkit.runtime.TelemetryThreadJVMTI.run(TelemetryThreadJVMTI.java:298)

 

"Signal Dispatcher" daemon prio=10 tid=0x00007fe7c0090000 nid=0x6369
runnable [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00007fe7c00b0000
nid=0x6368 waiting on condition [0x0000000000000000]

   java.lang.Thread.State: RUNNABLE

 

"Finalizer" daemon prio=10 tid=0x00007fe7c0067000 nid=0x6367 in
Object.wait() [0x00007fe7c71fb000]

   java.lang.Thread.State: WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7cc00ad28> (a
java.lang.ref.ReferenceQueue$Lock)

            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)

            - locked <0x00007fe7cc00ad28> (a
java.lang.ref.ReferenceQueue$Lock)

            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)

            at
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

 

"Reference Handler" daemon prio=10 tid=0x00007fe7c0040000 nid=0x6366 in
Object.wait() [0x00007fe7c72fc000]

   java.lang.Thread.State: WAITING (on object monitor)

            at java.lang.Object.wait(Native Method)

            - waiting on <0x00007fe7cc00ad58> (a
java.lang.ref.Reference$Lock)

            at java.lang.Object.wait(Object.java:485)

            at
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)

            - locked <0x00007fe7cc00ad58> (a java.lang.ref.Reference$Lock)

 

"VM Thread" prio=10 tid=0x00007fe7c0050000 nid=0x6365 runnable 

 

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040173000
nid=0x6362 runnable 

 

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x0000000040175000
nid=0x6363 runnable 

 

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x0000000040248000 nid=0x6364
runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007fe7c00f3000 nid=0x6371 waiting
on condition 

 

JNI global references: 1304

 

 

Found one Java-level deadlock:

=============================

"IPC Server handler 9 on 60020":

  waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher),

  which is held by "IPC Server handler 7 on 60020"

"IPC Server handler 7 on 60020":

  waiting for ownable synchronizer 0x00007fe7cbb06228, (a
java.util.concurrent.locks.ReentrantLock$NonfairSync),

  which is held by "regionserver60020.cacheFlusher"

"regionserver60020.cacheFlusher":

  waiting to lock monitor 0x00000000409f3908 (object 0x00007fe7cbacbd48, a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher),

  which is held by "IPC Server handler 7 on 60020"

 

Java stack information for the threads listed above:

===================================================

"IPC Server handler 9 on 60020":

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

"IPC Server handler 7 on 60020":

            at sun.misc.Unsafe.$$YJP$$park(Native Method)

            - parking to wait for  <0x00007fe7cbb06228> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)

            at sun.misc.Unsafe.park(Unsafe.java)

            at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(
AbstractQueuedSynchronizer.java:747)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Abstract
QueuedSynchronizer.java:778)

            at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueued
Synchronizer.java:1114)

            at
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java
:186)

            at
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(M
emStoreFlusher.java:429)

            - locked <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at
org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
2558)

            at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:597)

            at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)

            at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)

"regionserver60020.cacheFlusher":

            at java.util.ResourceBundle.endLoading(ResourceBundle.java:1506)

            - waiting to lock <0x00007fe7cbacbd48> (a
org.apache.hadoop.hbase.regionserver.MemStoreFlusher)

            at java.util.ResourceBundle.findBundle(ResourceBundle.java:1379)

            at java.util.ResourceBundle.findBundle(ResourceBundle.java:1292)

            at
java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1234)

            at java.util.ResourceBundle.getBundle(ResourceBundle.java:832)

            at sun.util.resources.LocaleData$1.run(LocaleData.java:127)

            at java.security.AccessController.$$YJP$$doPrivileged(Native
Method)

            at
java.security.AccessController.doPrivileged(AccessController.java)

            at sun.util.resources.LocaleData.getBundle(LocaleData.java:125)

            at
sun.util.resources.LocaleData.getTimeZoneNames(LocaleData.java:97)

            at
sun.util.TimeZoneNameUtility.getBundle(TimeZoneNameUtility.java:115)

            at
sun.util.TimeZoneNameUtility.retrieveDisplayNames(TimeZoneNameUtility.java:8
0)

            at java.util.TimeZone.getDisplayNames(TimeZone.java:399)

            at java.util.TimeZone.getDisplayName(TimeZone.java:350)

            at java.util.Date.toString(Date.java:1025)

            at java.lang.String.valueOf(String.java:2826)

            at java.lang.StringBuilder.append(StringBuilder.java:115)

            at
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue$CompactionReque
st.toString(PriorityCompactionQueue.java:114)

            at java.lang.String.valueOf(String.java:2826)

            at java.lang.StringBuilder.append(StringBuilder.java:115)

            at
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.addToRegionsInQ
ueue(PriorityCompactionQueue.java:145)

            - locked <0x00007fe7ccabd258> (a java.util.HashMap)

            at
org.apache.hadoop.hbase.regionserver.PriorityCompactionQueue.add(PriorityCom
pactionQueue.java:188)

            at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(Co
mpactSplitThread.java:140)

            - locked <0x00007fe7cbaf08c8> (a
org.apache.hadoop.hbase.regionserver.CompactSplitThread)

            at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(Co
mpactSplitThread.java:118)

            - locked <0x00007fe7cbaf08c8> (a
org.apache.hadoop.hbase.regionserver.CompactSplitThread)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlu
sher.java:387)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressu
re(MemStoreFlusher.java:194)

            at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.jav
a:214)

 

Found 1 deadlock.

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message