hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Johannes Schaback <johannes.schab...@visual-meta.com>
Subject Re: single RegionServer stuck, causing cluster to hang
Date Fri, 22 Aug 2014 17:59:19 GMT
Hi Michael,

thank you for your quick reply!

I paste an entire debug dump of a stuck RegionServer, including stacks etc.
I guess we have little to hide considering our problem :)

Master status for r3c7s4.colo.vm,60020,1408707352291 as of Fri Aug 22
15:15:46 CEST 2014

Version Info:
===========================================================
HBase 0.98.5-hadoop2
Subversion file:///var/tmp/0.98.5RC0/hbase-0.98.5 -r Unknown
Compiled by apurtell on Mon Aug  4 23:58:06 PDT 2014
Hadoop 2.4.1
Subversion http://svn.apache.org/repos/asf/hadoop/common -r 1604318
Compiled by jenkins on 2014-06-21T05:43Z


Tasks:
===========================================================
Task: RpcServer.reader=1,port=60020
Status: WAITING:Waiting for a call
Running for 5950s

Task: RpcServer.reader=2,port=60020
Status: WAITING:Waiting for a call
Running for 5920s

Task: RpcServer.reader=3,port=60020
Status: WAITING:Waiting for a call
Running for 5880s

Task: RpcServer.reader=4,port=60020
Status: WAITING:Waiting for a call
Running for 5850s

Task: RpcServer.reader=5,port=60020
Status: WAITING:Waiting for a call
Running for 5802s

Task: RpcServer.reader=6,port=60020
Status: WAITING:Waiting for a call
Running for 5707s

Task: RpcServer.reader=7,port=60020
Status: WAITING:Waiting for a call
Running for 5707s

Task: RpcServer.reader=8,port=60020
Status: WAITING:Waiting for a call
Running for 5700s

Task: RpcServer.reader=9,port=60020
Status: WAITING:Waiting for a call
Running for 5697s

Task: RpcServer.reader=0,port=60020
Status: WAITING:Waiting for a call
Running for 5697s



Executors:
===========================================================
  Status for executor: Executor-2-RS_OPEN_META-r3c7s4:60020
  =======================================
  0 events queued, 0 running
  Status for executor: Executor-5-RS_LOG_REPLAY_OPS-r3c7s4:60020
  =======================================
  0 events queued, 0 running
  Status for executor: Executor-1-RS_OPEN_REGION-r3c7s4:60020
  =======================================
  0 events queued, 0 running
  Status for executor: Executor-4-RS_CLOSE_META-r3c7s4:60020
  =======================================
  0 events queued, 0 running
  Status for executor: Executor-3-RS_CLOSE_REGION-r3c7s4:60020
  =======================================
  0 events queued, 0 running


Stacks:
===========================================================
Process Thread Dump:
85 active threads
Thread 2120 (1878278300@qtp-1123595403-7):
  State: RUNNABLE
  Blocked count: 887
  Waited count: 890
  Stack:
    sun.management.ThreadImpl.getThreadInfo1(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:174)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:139)

org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:165)

org.apache.hadoop.hbase.regionserver.RSDumpServlet.doGet(RSDumpServlet.java:81)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)

org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)

org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1089)

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)

org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)

org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
Thread 1609 (RS_CLOSE_REGION-r3c7s4:60020-2):
  State: WAITING
  Blocked count: 254
  Waited count: 406
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dd5615b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 1607 (RS_CLOSE_REGION-r3c7s4:60020-1):
  State: WAITING
  Blocked count: 210
  Waited count: 343
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dd5615b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 1604 (RS_CLOSE_REGION-r3c7s4:60020-0):
  State: WAITING
  Blocked count: 233
  Waited count: 374
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7dd5615b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 1355 (ResponseProcessor for block
BP-1157637685-192.168.3.192-1382642140917:blk_1294960049_255276569):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)

org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
    java.io.FilterInputStream.read(FilterInputStream.java:83)
    java.io.FilterInputStream.read(FilterInputStream.java:83)

org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1984)

org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)

org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:796)
Thread 1354 (DataStreamer for file
/hbase/WALs/r3c7s4.colo.vm,60020,1408707352291/r3c7s4.colo.vm%2C60020%2C1408707352291.1408710955280
block BP-1157637685-192.168.3.192-1382642140917:blk_1294960049_255276569):
  State: TIMED_WAITING
  Blocked count: 1061
  Waited count: 1088
  Stack:
    java.lang.Object.wait(Native Method)

org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:502)
Thread 181 (ReplicationExecutor-0):
  State: WAITING
  Blocked count: 15
  Waited count: 45
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18a8507c
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 144 (regionserver60020-splits-1408707430778):
  State: WAITING
  Blocked count: 39
  Waited count: 53
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@46b1c7f4
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 131 (regionserver60020-smallCompactions-1408707426607):
  State: WAITING
  Blocked count: 923
  Waited count: 1365
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@41be886e
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:539)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 122 (RS_OPEN_REGION-r3c7s4:60020-1):
  State: WAITING
  Blocked count: 1493
  Waited count: 2245
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43ef0f7a
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 123 (RS_OPEN_REGION-r3c7s4:60020-2):
  State: WAITING
  Blocked count: 1520
  Waited count: 2305
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43ef0f7a
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 121 (RS_OPEN_REGION-r3c7s4:60020-0):
  State: WAITING
  Blocked count: 1529
  Waited count: 2317
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43ef0f7a
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 120 (org.apache.hadoop.hdfs.PeerCache@51bd2431):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1973
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:245)
    org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41)
    org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119)
    java.lang.Thread.run(Thread.java:744)
Thread 116 (SplitLogWorker-r3c7s4.colo.vm,60020,1408707352291-EventThread):
  State: WAITING
  Blocked count: 0
  Waited count: 3
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3142a13
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Thread 115
(SplitLogWorker-r3c7s4.colo.vm,60020,1408707352291-SendThread(192.168.3.191:2181
)):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Thread 113 (SplitLogWorker-r3c7s4.colo.vm,60020,1408707352291):
  State: TIMED_WAITING
  Blocked count: 5
  Waited count: 1203
  Stack:
    java.lang.Object.wait(Native Method)

org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:261)

org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:205)
    java.lang.Thread.run(Thread.java:744)
Thread 112 (ReplicationRpcServer.handler=2,queue=0,port=60020):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53cd9646
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 111 (ReplicationRpcServer.handler=1,queue=0,port=60020):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53cd9646
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 110 (ReplicationRpcServer.handler=0,queue=0,port=60020):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53cd9646
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 109 (PriorityRpcServer.handler=9,queue=0,port=60020):
  State: WAITING
  Blocked count: 56
  Waited count: 159
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 108 (PriorityRpcServer.handler=8,queue=0,port=60020):
  State: WAITING
  Blocked count: 70
  Waited count: 180
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 107 (PriorityRpcServer.handler=7,queue=0,port=60020):
  State: WAITING
  Blocked count: 65
  Waited count: 170
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 106 (PriorityRpcServer.handler=6,queue=0,port=60020):
  State: WAITING
  Blocked count: 38
  Waited count: 145
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 105 (PriorityRpcServer.handler=5,queue=0,port=60020):
  State: WAITING
  Blocked count: 86
  Waited count: 194
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 104 (PriorityRpcServer.handler=4,queue=0,port=60020):
  State: WAITING
  Blocked count: 81
  Waited count: 193
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 103 (PriorityRpcServer.handler=3,queue=0,port=60020):
  State: WAITING
  Blocked count: 76
  Waited count: 167
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 102 (PriorityRpcServer.handler=2,queue=0,port=60020):
  State: WAITING
  Blocked count: 92
  Waited count: 171
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 101 (PriorityRpcServer.handler=1,queue=0,port=60020):
  State: WAITING
  Blocked count: 132
  Waited count: 207
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 100 (PriorityRpcServer.handler=0,queue=0,port=60020):
  State: WAITING
  Blocked count: 55
  Waited count: 153
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e44f8a5
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 97 (defaultRpcServer.handler=27,queue=0,port=60020):
  State: WAITING
  Blocked count: 176
  Waited count: 3136
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 94 (defaultRpcServer.handler=24,queue=0,port=60020):
  State: WAITING
  Blocked count: 147
  Waited count: 3093
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 91 (defaultRpcServer.handler=21,queue=0,port=60020):
  State: WAITING
  Blocked count: 163
  Waited count: 3107
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 90 (defaultRpcServer.handler=20,queue=2,port=60020):
  State: WAITING
  Blocked count: 227
  Waited count: 3792
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22048fd
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 88 (defaultRpcServer.handler=18,queue=0,port=60020):
  State: WAITING
  Blocked count: 175
  Waited count: 3104
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 82 (defaultRpcServer.handler=12,queue=0,port=60020):
  State: WAITING
  Blocked count: 173
  Waited count: 3131
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 81 (defaultRpcServer.handler=11,queue=2,port=60020):
  State: WAITING
  Blocked count: 214
  Waited count: 3832
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22048fd
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 76 (defaultRpcServer.handler=6,queue=0,port=60020):
  State: WAITING
  Blocked count: 208
  Waited count: 3171
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@24b3a041
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:111)
    org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
    java.lang.Thread.run(Thread.java:744)
Thread 12 (RpcServer.listener,port=60020):
  State: BLOCKED
  Blocked count: 2889
  Waited count: 0
  Blocked on org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
  Blocked by 14 (RpcServer.reader=1,port=60020)
  Stack:

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
    org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
Thread 24 (RpcServer.responder):
  State: RUNNABLE
  Blocked count: 159
  Waited count: 149
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.hadoop.hbase.ipc.RpcServer$Responder.doRunLoop(RpcServer.java:874)
    org.apache.hadoop.hbase.ipc.RpcServer$Responder.run(RpcServer.java:857)
Thread 69 (r3c7s4:60020Replication Statistics #0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 20
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 67 (regionserver60020-EventThread):
  State: WAITING
  Blocked count: 0
  Waited count: 3
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a92786f
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Thread 66 (regionserver60020-SendThread(192.168.3.191:2181)):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Thread 42 (regionserver60020.leaseChecker):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 600
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:95)
    java.lang.Thread.run(Thread.java:744)
Thread 44 (regionserver60020.nonceCleaner):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 17
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
    org.apache.hadoop.hbase.Chore.run(Chore.java:88)
    java.lang.Thread.run(Thread.java:744)
Thread 41 (regionserver60020.periodicFlusher):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 600
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
    org.apache.hadoop.hbase.Chore.run(Chore.java:88)
    java.lang.Thread.run(Thread.java:744)
Thread 40 (regionserver60020.compactionChecker):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 600
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
    org.apache.hadoop.hbase.Chore.run(Chore.java:88)
    java.lang.Thread.run(Thread.java:744)
Thread 63 (MemStoreFlusher.1):
  State: TIMED_WAITING
  Blocked count: 124
  Waited count: 828
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
    java.util.concurrent.DelayQueue.poll(DelayQueue.java:256)
    java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)

org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:233)
    java.lang.Thread.run(Thread.java:744)
Thread 61 (MemStoreFlusher.0):
  State: TIMED_WAITING
  Blocked count: 87
  Waited count: 775
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
    java.util.concurrent.DelayQueue.poll(DelayQueue.java:256)
    java.util.concurrent.DelayQueue.poll(DelayQueue.java:68)

org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:233)
    java.lang.Thread.run(Thread.java:744)
Thread 49 (regionserver60020.logRoller):
  State: TIMED_WAITING
  Blocked count: 11
  Waited count: 616
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78)
    java.lang.Thread.run(Thread.java:744)
Thread 60 (HBase-Metrics2-1):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 22808
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 59 (regionserver60020-WAL.AsyncNotifier):
  State: WAITING
  Blocked count: 2737
  Waited count: 2738
  Waiting on java.lang.Object@22104a3e
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)

org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncNotifier.run(FSHLog.java:1319)
    java.lang.Thread.run(Thread.java:744)
Thread 58 (regionserver60020-WAL.AsyncSyncer4):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.lang.Object@5b7a3dea
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)

org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
    java.lang.Thread.run(Thread.java:744)
Thread 57 (regionserver60020-WAL.AsyncSyncer3):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.lang.Object@3676fc91
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)

org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
    java.lang.Thread.run(Thread.java:744)
Thread 56 (regionserver60020-WAL.AsyncSyncer2):
  State: WAITING
  Blocked count: 5
  Waited count: 6
  Waiting on java.lang.Object@3dbfaf86
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)

org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
    java.lang.Thread.run(Thread.java:744)
Thread 55 (regionserver60020-WAL.AsyncSyncer1):
  State: WAITING
  Blocked count: 48
  Waited count: 49
  Waiting on java.lang.Object@739ca615
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)

org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
    java.lang.Thread.run(Thread.java:744)
Thread 54 (regionserver60020-WAL.AsyncSyncer0):
  State: WAITING
  Blocked count: 8235
  Waited count: 8234
  Waiting on java.lang.Object@315be7ff
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)

org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncSyncer.run(FSHLog.java:1199)
    java.lang.Thread.run(Thread.java:744)
Thread 53 (regionserver60020-WAL.AsyncWriter):
  State: WAITING
  Blocked count: 2737
  Waited count: 2737
  Waiting on java.lang.Object@8dc2faa
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)

org.apache.hadoop.hbase.regionserver.wal.FSHLog$AsyncWriter.run(FSHLog.java:1106)
    java.lang.Thread.run(Thread.java:744)
Thread 51 (LeaseRenewer:visualmeta@visualmeta):
  State: TIMED_WAITING
  Blocked count: 199
  Waited count: 6380
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438)
    org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
    org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
    java.lang.Thread.run(Thread.java:744)
Thread 48 (IPC Parameter Sending Thread #0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 7996
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)

java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
    java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 46 (IPC Client (1773132304) connection to r2c3s1.colo.vm/
192.168.3.171:60000 from visualmeta):
  State: TIMED_WAITING
  Blocked count: 1996
  Waited count: 1996
  Stack:
    java.lang.Object.wait(Native Method)

org.apache.hadoop.hbase.ipc.RpcClient$Connection.waitForWork(RpcClient.java:678)
    org.apache.hadoop.hbase.ipc.RpcClient$Connection.run(RpcClient.java:726)
Thread 45 (JvmPauseMonitor):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 11953
  Stack:
    java.lang.Thread.sleep(Native Method)

org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159)
    java.lang.Thread.run(Thread.java:744)
Thread 37 (regionserver60020-EventThread):
  State: WAITING
  Blocked count: 0
  Waited count: 3
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@378c93c2
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Thread 36 (regionserver60020-SendThread(192.168.3.171:2181)):
  State: RUNNABLE
  Blocked count: 24
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Thread 34 (regionserver60020-EventThread):
  State: WAITING
  Blocked count: 936
  Waited count: 967
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6e38c20b
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
    org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
Thread 33 (regionserver60020-SendThread(192.168.3.171:2181)):
  State: RUNNABLE
  Blocked count: 348
  Waited count: 0
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
    org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Thread 32 (regionserver60020):
  State: TIMED_WAITING
  Blocked count: 2068
  Waited count: 4060
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:92)
    org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:56)

org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:900)
    java.lang.Thread.run(Thread.java:744)
Thread 31 (Timer-0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 200
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:552)
    java.util.TimerThread.run(Timer.java:505)
Thread 30 (1702920744@qtp-1123595403-1 - Acceptor0
SelectChannelConnector@0.0.0.0:60030):
  State: RUNNABLE
  Blocked count: 156
  Waited count: 1
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)

org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498)
    org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)

org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)

org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)

org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Thread 27 (LruStats #0):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 20
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)

java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 25 (main.LruBlockCache.EvictionThread):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread@39a19ad8
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)

org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:705)
    java.lang.Thread.run(Thread.java:744)
Thread 23 (Timer for 'HBase' metrics system):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 600
  Stack:
    java.lang.Object.wait(Native Method)
    java.util.TimerThread.mainLoop(Timer.java:552)
    java.util.TimerThread.run(Timer.java:505)
Thread 22 (RpcServer.reader=9,port=60020):
  State: WAITING
  Blocked count: 289
  Waited count: 290
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)

org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)

org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 21 (RpcServer.reader=8,port=60020):
  State: WAITING
  Blocked count: 289
  Waited count: 291
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)

org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)

org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 20 (RpcServer.reader=7,port=60020):
  State: RUNNABLE
  Blocked count: 289
  Waited count: 289
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:578)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 19 (RpcServer.reader=6,port=60020):
  State: RUNNABLE
  Blocked count: 289
  Waited count: 289
  Stack:
    sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
    sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
    sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:578)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 18 (RpcServer.reader=5,port=60020):
  State: WAITING
  Blocked count: 290
  Waited count: 290
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)

org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)

org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 17 (RpcServer.reader=4,port=60020):
  State: WAITING
  Blocked count: 289
  Waited count: 290
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)

org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)

org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 16 (RpcServer.reader=3,port=60020):
  State: WAITING
  Blocked count: 289
  Waited count: 290
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)

org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)

org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 15 (RpcServer.reader=2,port=60020):
  State: WAITING
  Blocked count: 289
  Waited count: 290
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)

org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)

org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 14 (RpcServer.reader=1,port=60020):
  State: WAITING
  Blocked count: 289
  Waited count: 290
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)

org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)

org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 13 (RpcServer.reader=0,port=60020):
  State: WAITING
  Blocked count: 289
  Waited count: 290
  Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@40e06e28
  Stack:
    sun.misc.Unsafe.park(Native Method)
    java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)

java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:349)

org.apache.hadoop.hbase.ipc.BalancedQueueRpcExecutor.dispatch(BalancedQueueRpcExecutor.java:67)

org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.dispatch(SimpleRpcScheduler.java:129)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processRequest(RpcServer.java:1747)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.processOneRpc(RpcServer.java:1651)

org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1511)

org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:798)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:589)

org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:564)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    java.lang.Thread.run(Thread.java:744)
Thread 5 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 442
  Waited count: 107
  Waiting on java.lang.ref.ReferenceQueue$Lock@3c3c270c
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 131
  Waited count: 112
  Waiting on java.lang.ref.Reference$Lock@1127a3b9
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:503)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Thread 1 (main):
  State: WAITING
  Blocked count: 3
  Waited count: 2
  Waiting on java.lang.Thread@20bcf877
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Thread.join(Thread.java:1280)
    java.lang.Thread.join(Thread.java:1354)

org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64)

org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:85)
    org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2422)


RS Configuration:
===========================================================
<?xml version="1.0" encoding="UTF-8" standalone="no"?><configuration>
[snipp] no fancy stuff, all default, except absolute necessary settings
[snipp]
</configuration>

Logs
===========================================================
+++++++++++++++++++++++++++++++
/var/vm/apps/hbase-0.98.5-hadoop2/bin/../logs/hbase-visualmeta-regionserver-r3c7s4.log
+++++++++++++++++++++++++++++++
2014-08-22 14:52:46,121 INFO
 [PriorityRpcServer.handler=4,queue=0,port=60020]
regionserver.HRegionServer: Close a07ecf6735212703fe83a93678a86225, via
zk=yes, znode version=0, on r3c3s4.colo.vm,60020,1408711455393
2014-08-22 14:52:46,131 INFO
 [StoreCloserThread-image,nl\x00\x00\x00\x005\x1F\x1BN,1408356163124.682b252fdde4bb3bec3613e53bc9bec9.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,131 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,nl\x00\x00\x00\x005\x1F\x1BN,1408356163124.682b252fdde4bb3bec3613e53bc9bec9.
2014-08-22 14:52:46,131 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
682b252fdde4bb3bec3613e53bc9bec9 to
r3c2s1.colo.vm,60020,1408711451959:60020 as of 857256765
2014-08-22 14:52:46,156 INFO
 [PriorityRpcServer.handler=6,queue=0,port=60020]
regionserver.HRegionServer: Close 2dd5d2212b5e767aa2823c3af78b2ca7, via
zk=yes, znode version=0, on r2c9s4.colo.vm,60020,1408711451080
2014-08-22 14:52:46,161 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02\x82\xA2z\xA3,1408191336257.a07ecf6735212703fe83a93678a86225.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,162 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02\x82\xA2z\xA3,1408191336257.a07ecf6735212703fe83a93678a86225.
2014-08-22 14:52:46,162 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
a07ecf6735212703fe83a93678a86225 to
r3c3s4.colo.vm,60020,1408711455393:60020 as of 859701098
2014-08-22 14:52:46,174 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Close db0fae8af8fcd271f2911387327df40a, via
zk=yes, znode version=0, on r2c10s2.colo.vm,60020,1408711444387
2014-08-22 14:52:46,183 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02\x82\xAD\x9F],1408195161494.2dd5d2212b5e767aa2823c3af78b2ca7.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,183 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02\x82\xAD\x9F],1408195161494.2dd5d2212b5e767aa2823c3af78b2ca7.
2014-08-22 14:52:46,183 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
2dd5d2212b5e767aa2823c3af78b2ca7 to
r2c9s4.colo.vm,60020,1408711451080:60020 as of 860177691
2014-08-22 14:52:46,192 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Close e786e7c447b1614e69d46418c685a2cc, via
zk=yes, znode version=0, on r2c9s3.colo.vm,60020,1408711449726
2014-08-22 14:52:46,208 INFO
 [StoreCloserThread-image,be\x00\x00\x00\x00\x01xK%,1397196499096.db0fae8af8fcd271f2911387327df40a.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,208 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,be\x00\x00\x00\x00\x01xK%,1397196499096.db0fae8af8fcd271f2911387327df40a.
2014-08-22 14:52:46,208 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
db0fae8af8fcd271f2911387327df40a to
r2c10s2.colo.vm,60020,1408711444387:60020 as of 847254032
2014-08-22 14:52:46,228 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Close 72fb360847f0aa83ade3e54ca9061d5d, via
zk=yes, znode version=0, on r3c5s1.colo.vm,60020,1408711460148
2014-08-22 14:52:46,236 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02G\x06\xF2\xF6,1399131972275.e786e7c447b1614e69d46418c685a2cc.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,236 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02G\x06\xF2\xF6,1399131972275.e786e7c447b1614e69d46418c685a2cc.
2014-08-22 14:52:46,236 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
e786e7c447b1614e69d46418c685a2cc to
r2c9s3.colo.vm,60020,1408711449726:60020 as of 821132653
2014-08-22 14:52:46,246 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Close 1335c0778d21018fb0c8c4c54f9213c1, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408711481735
2014-08-22 14:52:46,261 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02}`}\xD6,1406846742363.72fb360847f0aa83ade3e54ca9061d5d.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,261 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02}`}\xD6,1406846742363.72fb360847f0aa83ade3e54ca9061d5d.
2014-08-22 14:52:46,261 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
72fb360847f0aa83ade3e54ca9061d5d to
r3c5s1.colo.vm,60020,1408711460148:60020 as of 809285789
2014-08-22 14:52:46,277 INFO
 [PriorityRpcServer.handler=8,queue=0,port=60020]
regionserver.HRegionServer: Close bba160a46f0a581c5c7cf9360e167d7e, via
zk=yes, znode version=0, on r2c9s4.colo.vm,60020,1408711451080
2014-08-22 14:52:46,278 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02w\xCAn\x0A,1405579293102.1335c0778d21018fb0c8c4c54f9213c1.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,278 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02w\xCAn\x0A,1405579293102.1335c0778d21018fb0c8c4c54f9213c1.
2014-08-22 14:52:46,278 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
1335c0778d21018fb0c8c4c54f9213c1 to
r3c5s3.colo.vm,60020,1408711481735:60020 as of 826458371
2014-08-22 14:52:46,296 INFO
 [PriorityRpcServer.handler=0,queue=0,port=60020]
regionserver.HRegionServer: Close 3ad114f3546150f1b2caee963e7d2618, via
zk=yes, znode version=0, on r3c7s2.colo.vm,60020,1408711484277
2014-08-22 14:52:46,313 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02\x82\x03\x05M,1408069202259.bba160a46f0a581c5c7cf9360e167d7e.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,313 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02\x82\x03\x05M,1408069202259.bba160a46f0a581c5c7cf9360e167d7e.
2014-08-22 14:52:46,313 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
bba160a46f0a581c5c7cf9360e167d7e to
r2c9s4.colo.vm,60020,1408711451080:60020 as of 857601277
2014-08-22 14:52:46,353 INFO
 [PriorityRpcServer.handler=2,queue=0,port=60020]
regionserver.HRegionServer: Close 14372e73c33d9b8c9848519ad628123e, via
zk=yes, znode version=0, on r3c2s1.colo.vm,60020,1408711451959
2014-08-22 14:52:46,357 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02Gq\xBF\xCC,1399165548009.3ad114f3546150f1b2caee963e7d2618.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,358 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02Gq\xBF\xCC,1399165548009.3ad114f3546150f1b2caee963e7d2618.
2014-08-22 14:52:46,358 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
3ad114f3546150f1b2caee963e7d2618 to
r3c7s2.colo.vm,60020,1408711484277:60020 as of 826418599
2014-08-22 14:52:46,395 INFO
 [PriorityRpcServer.handler=5,queue=0,port=60020]
regionserver.HRegionServer: Close 23ba7e5b9354d05a71c526d2fa0ecd47, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408711481735
2014-08-22 14:52:46,408 INFO
 [StoreCloserThread-image,it\x00\x00\x00\x00;\xBC\x18\x98,1405232618304.14372e73c33d9b8c9848519ad628123e.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,408 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,it\x00\x00\x00\x00;\xBC\x18\x98,1405232618304.14372e73c33d9b8c9848519ad628123e.
2014-08-22 14:52:46,409 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
14372e73c33d9b8c9848519ad628123e to
r3c2s1.colo.vm,60020,1408711451959:60020 as of 857293958
2014-08-22 14:52:46,410 INFO
 [StoreCloserThread-image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,411 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 14:52:46,411 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
23ba7e5b9354d05a71c526d2fa0ecd47 to
r3c5s3.colo.vm,60020,1408711481735:60020 as of 755730346
2014-08-22 14:52:46,419 INFO
 [PriorityRpcServer.handler=4,queue=0,port=60020]
regionserver.HRegionServer: Close b26a52eeb129185aa3e2b7f5c6c7268c, via
zk=yes, znode version=0, on r2c10s2.colo.vm,60020,1408711444387
2014-08-22 14:52:46,450 INFO
 [PriorityRpcServer.handler=6,queue=0,port=60020]
regionserver.HRegionServer: Close 9e6746f82fb374f0d74d57c1edcd8b79, via
zk=yes, znode version=0, on r3c7s3.colo.vm,60020,1408711499973
2014-08-22 14:52:46,459 INFO
 [StoreCloserThread-image,fi\x00\x00\x00\x00\x09\xDC\x80o,1398909121598.b26a52eeb129185aa3e2b7f5c6c7268c.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,459 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,fi\x00\x00\x00\x00\x09\xDC\x80o,1398909121598.b26a52eeb129185aa3e2b7f5c6c7268c.
2014-08-22 14:52:46,459 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
b26a52eeb129185aa3e2b7f5c6c7268c to
r2c10s2.colo.vm,60020,1408711444387:60020 as of 800831945
2014-08-22 14:52:46,482 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02D\x0F\xE9\x94,1398936500039.9e6746f82fb374f0d74d57c1edcd8b79.-1]
regionserver.HStore: Closed cf
2014-08-22 14:52:46,482 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02D\x0F\xE9\x94,1398936500039.9e6746f82fb374f0d74d57c1edcd8b79.
2014-08-22 14:52:46,482 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
9e6746f82fb374f0d74d57c1edcd8b79 to
r3c7s3.colo.vm,60020,1408711499973:60020 as of 824230259
2014-08-22 14:52:55,284 INFO  [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
item,fr\x00\x00\x00\x00\x01\x14\xFB\xC6,1408559156558.357acd681a22c550ca1696dbe1e65b15.
after a delay of 17900
2014-08-22 14:53:05,284 INFO  [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
item,fr\x00\x00\x00\x00\x01\x14\xFB\xC6,1408559156558.357acd681a22c550ca1696dbe1e65b15.
after a delay of 20873
2014-08-22 14:53:13,304 INFO  [MemStoreFlusher.0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=857612072,
memsize=18.4 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/item/357acd681a22c550ca1696dbe1e65b15/.tmp/3bb4beaea04c497d9fa4410247557ac3
2014-08-22 14:53:13,324 INFO  [MemStoreFlusher.0] regionserver.HStore:
Added
hdfs://visualmeta/hbase/data/default/item/357acd681a22c550ca1696dbe1e65b15/cf_feature_vector/3bb4beaea04c497d9fa4410247557ac3,
entries=100, sequenceid=857612072, filesize=6.8 K
2014-08-22 14:53:13,325 INFO  [MemStoreFlusher.0] regionserver.HRegion:
Finished memstore flush of ~18.4 K/18800, currentsize=0/0 for region
item,fr\x00\x00\x00\x00\x01\x14\xFB\xC6,1408559156558.357acd681a22c550ca1696dbe1e65b15.
in 141ms, sequenceid=857612072, compaction requested=true
2014-08-22 14:54:05,284 INFO  [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
after a delay of 12130
2014-08-22 14:54:05,285 INFO  [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. after
a delay of 14728
2014-08-22 14:54:15,284 INFO  [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
after a delay of 19426
2014-08-22 14:54:15,284 INFO  [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. after
a delay of 10422
2014-08-22 14:54:17,490 INFO  [MemStoreFlusher.0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=830369741,
memsize=337.9 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/classifier/3129db940c9f94729b986b742f0161c2/.tmp/0d82d522fcd14c00bd91a8a964e7decf
2014-08-22 14:54:17,511 INFO  [MemStoreFlusher.0] regionserver.HStore:
Added
hdfs://visualmeta/hbase/data/default/classifier/3129db940c9f94729b986b742f0161c2/cf/0d82d522fcd14c00bd91a8a964e7decf,
entries=5, sequenceid=830369741, filesize=338.4 K
2014-08-22 14:54:17,511 INFO  [MemStoreFlusher.0] regionserver.HRegion:
Finished memstore flush of ~337.9 K/346008, currentsize=0/0 for region
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
in 97ms, sequenceid=830369741, compaction requested=true
2014-08-22 14:54:20,124 INFO  [MemStoreFlusher.1]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=857453886,
memsize=661.7 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/classifier/82f8098871398b59ead1c6963781685d/.tmp/d28487d6829e400c87879b1e666e469c
2014-08-22 14:54:20,199 INFO  [MemStoreFlusher.1] regionserver.HStore:
Added
hdfs://visualmeta/hbase/data/default/classifier/82f8098871398b59ead1c6963781685d/cf/d28487d6829e400c87879b1e666e469c,
entries=7, sequenceid=857453886, filesize=662.1 K
2014-08-22 14:54:20,199 INFO  [MemStoreFlusher.1] regionserver.HRegion:
Finished memstore flush of ~661.7 K/677544, currentsize=0/0 for region
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. in
186ms, sequenceid=857453886, compaction requested=true
2014-08-22 14:54:20,200 INFO
 [regionserver60020-smallCompactions-1408707426607] regionserver.HRegion:
Starting compaction on cf in region
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d.
2014-08-22 14:54:20,200 INFO
 [regionserver60020-smallCompactions-1408707426607] regionserver.HStore:
Starting compaction of 3 file(s) in cf of
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. into
tmpdir=hdfs://visualmeta/hbase/data/default/classifier/82f8098871398b59ead1c6963781685d/.tmp,
totalSize=46.1 M
2014-08-22 14:54:21,242 INFO
 [regionserver60020-smallCompactions-1408707426607] regionserver.HStore:
Completed compaction of 3 file(s) in cf of
classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d. into
22f1215d31f34490aaf2465a0414c16f(size=45.4 M), total size for store is
633.1 M. This selection was in queue for 0sec, and took 1sec to execute.
2014-08-22 14:54:21,242 INFO
 [regionserver60020-smallCompactions-1408707426607]
regionserver.CompactSplitThread: Completed compaction: Request =
regionName=classifier,%\xFD\xA3,1401198483217.82f8098871398b59ead1c6963781685d.,
storeName=cf, fileCount=3, fileSize=46.1 M, priority=5,
time=2525894161984378; duration=1sec
2014-08-22 15:00:53,222 INFO  [regionserver60020-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r1c14s3.colo.vm,60020,1408688739222 znode expired, triggering
replicatorRemoved event
2014-08-22 15:00:53,744 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
2014-08-22 15:00:53,769 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02Q\xA4\xA4\xCF,1400622870851.0606a108cfb00a6954eae8c84bc39669.
2014-08-22 15:00:53,829 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02{\x04>=,1406099001127.7d62f7f7139aea672d70e0e78f6edfda.
2014-08-22 15:00:53,859 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,dk\x00\x00\x00\x00u\xB5\xB4\x1E,1405615660019.02aa0832f54f795aad7f4016a3a87a99.
2014-08-22 15:00:53,885 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fi\x00\x00\x00\x00\x0A~\xFA\x08,1400494510921.2b16fc52a132709296646ebc00b2f616.
2014-08-22 15:00:53,900 INFO
 [StoreOpener-c4b8745f4427e86ec1668eabe6adea1a-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:53,901 INFO
 [StoreOpener-0606a108cfb00a6954eae8c84bc39669-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:53,911 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for cd296eb45a6e42698057e83c21c8a44c
2014-08-22 15:00:53,928 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for cf2bbb2640d549d0aaab73150303c9af
2014-08-22 15:00:53,929 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined c4b8745f4427e86ec1668eabe6adea1a; next
sequenceid=854334330
2014-08-22 15:00:53,935 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 0606a108cfb00a6954eae8c84bc39669; next
sequenceid=826450913
2014-08-22 15:00:53,980 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\xA4x6\x0B,1405342105031.f4de800d88fb4bd41eb746f9d4b747be.
2014-08-22 15:00:53,983 INFO
 [StoreOpener-7d62f7f7139aea672d70e0e78f6edfda-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:53,987 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 7423e1fa193a4ee9858acc0b44d5cabd
2014-08-22 15:00:53,989 INFO
 [PostOpenDeployTasks:c4b8745f4427e86ec1668eabe6adea1a]
regionserver.HRegionServer: Post open deploy tasks for
region=classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
2014-08-22 15:00:53,989 INFO
 [PostOpenDeployTasks:0606a108cfb00a6954eae8c84bc39669]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02Q\xA4\xA4\xCF,1400622870851.0606a108cfb00a6954eae8c84bc39669.
2014-08-22 15:00:53,989 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,it\x00\x00\x00\x006]x\x82,1401401129195.a440630ea65a477b027cf6dcfcaca548.
2014-08-22 15:00:53,989 INFO
 [regionserver60020-smallCompactions-1408707426607] regionserver.HRegion:
Starting compaction on cf in region
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
2014-08-22 15:00:53,990 INFO
 [regionserver60020-smallCompactions-1408707426607] regionserver.HStore:
Starting compaction of 3 file(s) in cf of
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
into
tmpdir=hdfs://visualmeta/hbase/data/default/classifier/c4b8745f4427e86ec1668eabe6adea1a/.tmp,
totalSize=997.9 K
2014-08-22 15:00:54,016 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,pl\x00\x00\x00\x00\x06\x0A\x0A\xB1,1404864856073.e8f628a9e05b5c425e1bceb90e407016.
2014-08-22 15:00:54,020 INFO
 [PostOpenDeployTasks:c4b8745f4427e86ec1668eabe6adea1a] catalog.MetaEditor:
Updated row
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,020 INFO
 [PostOpenDeployTasks:c4b8745f4427e86ec1668eabe6adea1a]
regionserver.HRegionServer: Finished post open deploy task for
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
2014-08-22 15:00:54,020 INFO
 [PostOpenDeployTasks:0606a108cfb00a6954eae8c84bc39669] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02Q\xA4\xA4\xCF,1400622870851.0606a108cfb00a6954eae8c84bc39669.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,021 INFO
 [PostOpenDeployTasks:0606a108cfb00a6954eae8c84bc39669]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02Q\xA4\xA4\xCF,1400622870851.0606a108cfb00a6954eae8c84bc39669.
2014-08-22 15:00:54,038 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Open
image,se\x00\x00\x00\x00-\x8C\xD2},1407897158298.deabfc61edb34a1c8ee5366a467f15cd.
2014-08-22 15:00:54,059 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for fd14404a2b524d99a7584a1847f6d453
2014-08-22 15:00:54,072 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 7d62f7f7139aea672d70e0e78f6edfda; next
sequenceid=821164840
2014-08-22 15:00:54,117 INFO
 [PostOpenDeployTasks:7d62f7f7139aea672d70e0e78f6edfda]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02{\x04>=,1406099001127.7d62f7f7139aea672d70e0e78f6edfda.
2014-08-22 15:00:54,151 INFO
 [StoreOpener-02aa0832f54f795aad7f4016a3a87a99-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,166 INFO
 [PostOpenDeployTasks:7d62f7f7139aea672d70e0e78f6edfda] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02{\x04>=,1406099001127.7d62f7f7139aea672d70e0e78f6edfda.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,166 INFO
 [PostOpenDeployTasks:7d62f7f7139aea672d70e0e78f6edfda]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02{\x04>=,1406099001127.7d62f7f7139aea672d70e0e78f6edfda.
2014-08-22 15:00:54,170 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 50040f72f62e49909e9049218b1ca092
2014-08-22 15:00:54,183 INFO
 [StoreOpener-2b16fc52a132709296646ebc00b2f616-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,184 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 9941517d71e94f85ba721053ff58d8d1
2014-08-22 15:00:54,193 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 5c668fc88da548c4873300850a53837d
2014-08-22 15:00:54,195 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 2b16fc52a132709296646ebc00b2f616; next
sequenceid=824869877
2014-08-22 15:00:54,210 INFO
 [PostOpenDeployTasks:2b16fc52a132709296646ebc00b2f616]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fi\x00\x00\x00\x00\x0A~\xFA\x08,1400494510921.2b16fc52a132709296646ebc00b2f616.
2014-08-22 15:00:54,223 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 02aa0832f54f795aad7f4016a3a87a99; next
sequenceid=826595682
2014-08-22 15:00:54,232 INFO
 [PostOpenDeployTasks:02aa0832f54f795aad7f4016a3a87a99]
regionserver.HRegionServer: Post open deploy tasks for
region=image,dk\x00\x00\x00\x00u\xB5\xB4\x1E,1405615660019.02aa0832f54f795aad7f4016a3a87a99.
2014-08-22 15:00:54,234 INFO
 [regionserver60020-smallCompactions-1408707426607] regionserver.HStore:
Completed compaction of 3 file(s) in cf of
classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.
into 4ce2bb9edbe240b1b439e433e4fcd352(size=916.2 K), total size for store
is 433.3 M. This selection was in queue for 0sec, and took 0sec to execute.
2014-08-22 15:00:54,234 INFO
 [regionserver60020-smallCompactions-1408707426607]
regionserver.CompactSplitThread: Completed compaction: Request =
regionName=classifier,\xCA\x90\xCD,1408632541341.c4b8745f4427e86ec1668eabe6adea1a.,
storeName=cf, fileCount=3, fileSize=997.9 K, priority=6,
time=2526287950921618; duration=0sec
2014-08-22 15:00:54,235 INFO
 [PostOpenDeployTasks:2b16fc52a132709296646ebc00b2f616] catalog.MetaEditor:
Updated row
image,fi\x00\x00\x00\x00\x0A~\xFA\x08,1400494510921.2b16fc52a132709296646ebc00b2f616.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,236 INFO
 [PostOpenDeployTasks:2b16fc52a132709296646ebc00b2f616]
regionserver.HRegionServer: Finished post open deploy task for
image,fi\x00\x00\x00\x00\x0A~\xFA\x08,1400494510921.2b16fc52a132709296646ebc00b2f616.
2014-08-22 15:00:54,269 INFO
 [StoreOpener-f4de800d88fb4bd41eb746f9d4b747be-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,269 INFO
 [PostOpenDeployTasks:02aa0832f54f795aad7f4016a3a87a99] catalog.MetaEditor:
Updated row
image,dk\x00\x00\x00\x00u\xB5\xB4\x1E,1405615660019.02aa0832f54f795aad7f4016a3a87a99.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,270 INFO
 [PostOpenDeployTasks:02aa0832f54f795aad7f4016a3a87a99]
regionserver.HRegionServer: Finished post open deploy task for
image,dk\x00\x00\x00\x00u\xB5\xB4\x1E,1405615660019.02aa0832f54f795aad7f4016a3a87a99.
2014-08-22 15:00:54,278 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for fe60d458f1644cb893c28b265407a8b6
2014-08-22 15:00:54,280 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined f4de800d88fb4bd41eb746f9d4b747be; next
sequenceid=827405090
2014-08-22 15:00:54,316 INFO
 [PostOpenDeployTasks:f4de800d88fb4bd41eb746f9d4b747be]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\xA4x6\x0B,1405342105031.f4de800d88fb4bd41eb746f9d4b747be.
2014-08-22 15:00:54,348 INFO
 [PostOpenDeployTasks:f4de800d88fb4bd41eb746f9d4b747be] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\xA4x6\x0B,1405342105031.f4de800d88fb4bd41eb746f9d4b747be.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,348 INFO
 [PostOpenDeployTasks:f4de800d88fb4bd41eb746f9d4b747be]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\xA4x6\x0B,1405342105031.f4de800d88fb4bd41eb746f9d4b747be.
2014-08-22 15:00:54,349 INFO
 [StoreOpener-a440630ea65a477b027cf6dcfcaca548-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,362 INFO
 [StoreOpener-e8f628a9e05b5c425e1bceb90e407016-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,364 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 1d0bd533c1de4667a7b3d56c003ed461
2014-08-22 15:00:54,366 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 63d6d479b0a54aa98cbcaff47c3cce26
2014-08-22 15:00:54,368 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for d77fad667b56456c95c58dd5ed8012a6
2014-08-22 15:00:54,392 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for ae9264312faf4d7f97f8d6d801ac648e
2014-08-22 15:00:54,394 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined e8f628a9e05b5c425e1bceb90e407016; next
sequenceid=826674750
2014-08-22 15:00:54,400 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined a440630ea65a477b027cf6dcfcaca548; next
sequenceid=848079895
2014-08-22 15:00:54,408 INFO
 [PostOpenDeployTasks:e8f628a9e05b5c425e1bceb90e407016]
regionserver.HRegionServer: Post open deploy tasks for
region=image,pl\x00\x00\x00\x00\x06\x0A\x0A\xB1,1404864856073.e8f628a9e05b5c425e1bceb90e407016.
2014-08-22 15:00:54,408 INFO
 [PostOpenDeployTasks:a440630ea65a477b027cf6dcfcaca548]
regionserver.HRegionServer: Post open deploy tasks for
region=image,it\x00\x00\x00\x006]x\x82,1401401129195.a440630ea65a477b027cf6dcfcaca548.
2014-08-22 15:00:54,417 INFO
 [StoreOpener-deabfc61edb34a1c8ee5366a467f15cd-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:00:54,425 INFO
 [PostOpenDeployTasks:e8f628a9e05b5c425e1bceb90e407016] catalog.MetaEditor:
Updated row
image,pl\x00\x00\x00\x00\x06\x0A\x0A\xB1,1404864856073.e8f628a9e05b5c425e1bceb90e407016.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,425 INFO
 [PostOpenDeployTasks:e8f628a9e05b5c425e1bceb90e407016]
regionserver.HRegionServer: Finished post open deploy task for
image,pl\x00\x00\x00\x00\x06\x0A\x0A\xB1,1404864856073.e8f628a9e05b5c425e1bceb90e407016.
2014-08-22 15:00:54,425 INFO
 [PostOpenDeployTasks:a440630ea65a477b027cf6dcfcaca548] catalog.MetaEditor:
Updated row
image,it\x00\x00\x00\x006]x\x82,1401401129195.a440630ea65a477b027cf6dcfcaca548.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,425 INFO
 [PostOpenDeployTasks:a440630ea65a477b027cf6dcfcaca548]
regionserver.HRegionServer: Finished post open deploy task for
image,it\x00\x00\x00\x006]x\x82,1401401129195.a440630ea65a477b027cf6dcfcaca548.
2014-08-22 15:00:54,441 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined deabfc61edb34a1c8ee5366a467f15cd; next
sequenceid=833150761
2014-08-22 15:00:54,455 INFO
 [PostOpenDeployTasks:deabfc61edb34a1c8ee5366a467f15cd]
regionserver.HRegionServer: Post open deploy tasks for
region=image,se\x00\x00\x00\x00-\x8C\xD2},1407897158298.deabfc61edb34a1c8ee5366a467f15cd.
2014-08-22 15:00:54,473 INFO
 [PostOpenDeployTasks:deabfc61edb34a1c8ee5366a467f15cd] catalog.MetaEditor:
Updated row
image,se\x00\x00\x00\x00-\x8C\xD2},1407897158298.deabfc61edb34a1c8ee5366a467f15cd.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:00:54,473 INFO
 [PostOpenDeployTasks:deabfc61edb34a1c8ee5366a467f15cd]
regionserver.HRegionServer: Finished post open deploy task for
image,se\x00\x00\x00\x00-\x8C\xD2},1407897158298.deabfc61edb34a1c8ee5366a467f15cd.
2014-08-22 15:00:55,627 INFO  [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Moving
r1c14s3.colo.vm,60020,1408688739222's hlogs to my queue
2014-08-22 15:00:55,637 INFO  [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Won't transfer the queue, another RS
took care of it because of: KeeperErrorCode = NoNode for
/hbase/replication/rs/r1c14s3.colo.vm,60020,1408688739222/lock
2014-08-22 15:02:58,116 INFO  [regionserver60020-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r3c3s4.colo.vm,60020,1408711455393 znode expired, triggering
replicatorRemoved event
2014-08-22 15:02:58,674 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open classifier,\xDD|
\x0B\xD6_X(,1404308250116.991ff9d5f42cba40ae71bcd17ea5000f.
2014-08-22 15:02:58,701 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02H\xEF6\xBE,1400132008394.d4d5cf8c8d1c64a00a235df670036e01.
2014-08-22 15:02:58,731 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02}\x1DfW,1406751533083.67dbe3fa51662bd6753df1e0c5314a92.
2014-08-22 15:02:58,746 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,es\x00\x00\x00\x00"Wfu,1402968397139.3de6d45475ae36aeb3cfa3b823ceaeba.
2014-08-22 15:02:58,749 INFO
 [StoreOpener-991ff9d5f42cba40ae71bcd17ea5000f-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,760 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 991ff9d5f42cba40ae71bcd17ea5000f; next
sequenceid=826516917
2014-08-22 15:02:58,773 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\x91\xA17\xFE,1398927754577.48c5d3d1a5b744f4d5d84bb2cd392986.
2014-08-22 15:02:58,773 INFO
 [PostOpenDeployTasks:991ff9d5f42cba40ae71bcd17ea5000f]
regionserver.HRegionServer: Post open deploy tasks for
region=classifier,\xDD|
\x0B\xD6_X(,1404308250116.991ff9d5f42cba40ae71bcd17ea5000f.
2014-08-22 15:02:58,775 INFO
 [StoreOpener-d4d5cf8c8d1c64a00a235df670036e01-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,780 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 02cf17c9c191440d8aac96205834569d
2014-08-22 15:02:58,791 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 9b2d695887454347b7c12a9edac977bc
2014-08-22 15:02:58,793 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined d4d5cf8c8d1c64a00a235df670036e01; next
sequenceid=847122540
2014-08-22 15:02:58,806 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\xAB\xE5\xD8\xA6,1408494911543.468e62d7a5e8eb41ed8fd2d25d5950b8.
2014-08-22 15:02:58,809 INFO
 [StoreOpener-67dbe3fa51662bd6753df1e0c5314a92-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,810 INFO
 [PostOpenDeployTasks:991ff9d5f42cba40ae71bcd17ea5000f] catalog.MetaEditor:
Updated row classifier,\xDD|
\x0B\xD6_X(,1404308250116.991ff9d5f42cba40ae71bcd17ea5000f. with
server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:58,810 INFO
 [PostOpenDeployTasks:991ff9d5f42cba40ae71bcd17ea5000f]
regionserver.HRegionServer: Finished post open deploy task for
classifier,\xDD| \x0B\xD6_X(,1404308250116.991ff9d5f42cba40ae71bcd17ea5000f.
2014-08-22 15:02:58,812 INFO
 [PostOpenDeployTasks:d4d5cf8c8d1c64a00a235df670036e01]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02H\xEF6\xBE,1400132008394.d4d5cf8c8d1c64a00a235df670036e01.
2014-08-22 15:02:58,820 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 346cc92277bb4813a58d509b8c00ccd9
2014-08-22 15:02:58,821 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 67dbe3fa51662bd6753df1e0c5314a92; next
sequenceid=826459200
2014-08-22 15:02:58,833 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,it\x00\x00\x00\x008\xD6g\x8A,1402990061104.1bb84f733bb61e56f6fab8f0368d1f18.
2014-08-22 15:02:58,833 INFO
 [PostOpenDeployTasks:67dbe3fa51662bd6753df1e0c5314a92]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02}\x1DfW,1406751533083.67dbe3fa51662bd6753df1e0c5314a92.
2014-08-22 15:02:58,837 INFO
 [PostOpenDeployTasks:d4d5cf8c8d1c64a00a235df670036e01] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02H\xEF6\xBE,1400132008394.d4d5cf8c8d1c64a00a235df670036e01.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:58,837 INFO
 [PostOpenDeployTasks:d4d5cf8c8d1c64a00a235df670036e01]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02H\xEF6\xBE,1400132008394.d4d5cf8c8d1c64a00a235df670036e01.
2014-08-22 15:02:58,863 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,nl\x00\x00\x00\x002\xD8\xA8$,1404928936490.21e6c12b729c76e144924f36f34cba4e.
2014-08-22 15:02:58,866 INFO
 [PostOpenDeployTasks:67dbe3fa51662bd6753df1e0c5314a92] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02}\x1DfW,1406751533083.67dbe3fa51662bd6753df1e0c5314a92.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:58,866 INFO
 [PostOpenDeployTasks:67dbe3fa51662bd6753df1e0c5314a92]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02}\x1DfW,1406751533083.67dbe3fa51662bd6753df1e0c5314a92.
2014-08-22 15:02:58,872 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,ru\x00\x00\x00\x00\x0250\xBD,1404692062662.4af7a8fb3cd560d92c074bc1fce01ae6.
2014-08-22 15:02:58,896 INFO
 [StoreOpener-3de6d45475ae36aeb3cfa3b823ceaeba-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,901 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 9fb48e6e462f49f290a0034c01661a11
2014-08-22 15:02:58,902 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Open
image,tr\x00\x00\x00\x00\x04\xB1f\x0C,1405919250032.60522dfa3cd1b4a2963c3a5aad56f06b.
2014-08-22 15:02:58,910 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 3de6d45475ae36aeb3cfa3b823ceaeba; next
sequenceid=826442691
2014-08-22 15:02:58,947 INFO
 [PostOpenDeployTasks:3de6d45475ae36aeb3cfa3b823ceaeba]
regionserver.HRegionServer: Post open deploy tasks for
region=image,es\x00\x00\x00\x00"Wfu,1402968397139.3de6d45475ae36aeb3cfa3b823ceaeba.
2014-08-22 15:02:58,950 INFO
 [StoreOpener-48c5d3d1a5b744f4d5d84bb2cd392986-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:58,958 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 48c5d3d1a5b744f4d5d84bb2cd392986; next
sequenceid=826605440
2014-08-22 15:02:58,980 INFO
 [PostOpenDeployTasks:48c5d3d1a5b744f4d5d84bb2cd392986]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\x91\xA17\xFE,1398927754577.48c5d3d1a5b744f4d5d84bb2cd392986.
2014-08-22 15:02:58,982 INFO
 [PostOpenDeployTasks:3de6d45475ae36aeb3cfa3b823ceaeba] catalog.MetaEditor:
Updated row
image,es\x00\x00\x00\x00"Wfu,1402968397139.3de6d45475ae36aeb3cfa3b823ceaeba.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:58,982 INFO
 [PostOpenDeployTasks:3de6d45475ae36aeb3cfa3b823ceaeba]
regionserver.HRegionServer: Finished post open deploy task for
image,es\x00\x00\x00\x00"Wfu,1402968397139.3de6d45475ae36aeb3cfa3b823ceaeba.
2014-08-22 15:02:58,983 INFO
 [StoreOpener-468e62d7a5e8eb41ed8fd2d25d5950b8-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,016 INFO
 [PostOpenDeployTasks:48c5d3d1a5b744f4d5d84bb2cd392986] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\x91\xA17\xFE,1398927754577.48c5d3d1a5b744f4d5d84bb2cd392986.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,016 INFO
 [PostOpenDeployTasks:48c5d3d1a5b744f4d5d84bb2cd392986]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\x91\xA17\xFE,1398927754577.48c5d3d1a5b744f4d5d84bb2cd392986.
2014-08-22 15:02:59,019 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 468e62d7a5e8eb41ed8fd2d25d5950b8; next
sequenceid=855595540
2014-08-22 15:02:59,046 INFO
 [PostOpenDeployTasks:468e62d7a5e8eb41ed8fd2d25d5950b8]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\xAB\xE5\xD8\xA6,1408494911543.468e62d7a5e8eb41ed8fd2d25d5950b8.
2014-08-22 15:02:59,076 INFO
 [PostOpenDeployTasks:468e62d7a5e8eb41ed8fd2d25d5950b8] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\xAB\xE5\xD8\xA6,1408494911543.468e62d7a5e8eb41ed8fd2d25d5950b8.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,076 INFO
 [PostOpenDeployTasks:468e62d7a5e8eb41ed8fd2d25d5950b8]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\xAB\xE5\xD8\xA6,1408494911543.468e62d7a5e8eb41ed8fd2d25d5950b8.
2014-08-22 15:02:59,085 INFO
 [StoreOpener-1bb84f733bb61e56f6fab8f0368d1f18-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,097 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for aabe8cf7bc0242fe9fc5a84cc6c9007f
2014-08-22 15:02:59,101 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for ce472d7873344b979a7228f40db343f9
2014-08-22 15:02:59,103 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 1bb84f733bb61e56f6fab8f0368d1f18; next
sequenceid=824267600
2014-08-22 15:02:59,113 INFO
 [PostOpenDeployTasks:1bb84f733bb61e56f6fab8f0368d1f18]
regionserver.HRegionServer: Post open deploy tasks for
region=image,it\x00\x00\x00\x008\xD6g\x8A,1402990061104.1bb84f733bb61e56f6fab8f0368d1f18.
2014-08-22 15:02:59,115 INFO
 [StoreOpener-21e6c12b729c76e144924f36f34cba4e-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,124 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 21e6c12b729c76e144924f36f34cba4e; next
sequenceid=824260645
2014-08-22 15:02:59,143 INFO
 [PostOpenDeployTasks:21e6c12b729c76e144924f36f34cba4e]
regionserver.HRegionServer: Post open deploy tasks for
region=image,nl\x00\x00\x00\x002\xD8\xA8$,1404928936490.21e6c12b729c76e144924f36f34cba4e.
2014-08-22 15:02:59,145 INFO
 [PostOpenDeployTasks:1bb84f733bb61e56f6fab8f0368d1f18] catalog.MetaEditor:
Updated row
image,it\x00\x00\x00\x008\xD6g\x8A,1402990061104.1bb84f733bb61e56f6fab8f0368d1f18.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,145 INFO
 [PostOpenDeployTasks:1bb84f733bb61e56f6fab8f0368d1f18]
regionserver.HRegionServer: Finished post open deploy task for
image,it\x00\x00\x00\x008\xD6g\x8A,1402990061104.1bb84f733bb61e56f6fab8f0368d1f18.
2014-08-22 15:02:59,175 INFO
 [StoreOpener-4af7a8fb3cd560d92c074bc1fce01ae6-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,175 INFO
 [PostOpenDeployTasks:21e6c12b729c76e144924f36f34cba4e] catalog.MetaEditor:
Updated row
image,nl\x00\x00\x00\x002\xD8\xA8$,1404928936490.21e6c12b729c76e144924f36f34cba4e.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,175 INFO
 [PostOpenDeployTasks:21e6c12b729c76e144924f36f34cba4e]
regionserver.HRegionServer: Finished post open deploy task for
image,nl\x00\x00\x00\x002\xD8\xA8$,1404928936490.21e6c12b729c76e144924f36f34cba4e.
2014-08-22 15:02:59,184 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 54b1f0ac77d04fb19b45c6d15f97e1e2
2014-08-22 15:02:59,189 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 4af7a8fb3cd560d92c074bc1fce01ae6; next
sequenceid=795162564
2014-08-22 15:02:59,209 INFO
 [PostOpenDeployTasks:4af7a8fb3cd560d92c074bc1fce01ae6]
regionserver.HRegionServer: Post open deploy tasks for
region=image,ru\x00\x00\x00\x00\x0250\xBD,1404692062662.4af7a8fb3cd560d92c074bc1fce01ae6.
2014-08-22 15:02:59,223 INFO
 [PostOpenDeployTasks:4af7a8fb3cd560d92c074bc1fce01ae6] catalog.MetaEditor:
Updated row
image,ru\x00\x00\x00\x00\x0250\xBD,1404692062662.4af7a8fb3cd560d92c074bc1fce01ae6.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,223 INFO
 [PostOpenDeployTasks:4af7a8fb3cd560d92c074bc1fce01ae6]
regionserver.HRegionServer: Finished post open deploy task for
image,ru\x00\x00\x00\x00\x0250\xBD,1404692062662.4af7a8fb3cd560d92c074bc1fce01ae6.
2014-08-22 15:02:59,223 INFO
 [StoreOpener-60522dfa3cd1b4a2963c3a5aad56f06b-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:02:59,241 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 60522dfa3cd1b4a2963c3a5aad56f06b; next
sequenceid=826442022
2014-08-22 15:02:59,251 INFO
 [PostOpenDeployTasks:60522dfa3cd1b4a2963c3a5aad56f06b]
regionserver.HRegionServer: Post open deploy tasks for
region=image,tr\x00\x00\x00\x00\x04\xB1f\x0C,1405919250032.60522dfa3cd1b4a2963c3a5aad56f06b.
2014-08-22 15:02:59,265 INFO
 [PostOpenDeployTasks:60522dfa3cd1b4a2963c3a5aad56f06b] catalog.MetaEditor:
Updated row
image,tr\x00\x00\x00\x00\x04\xB1f\x0C,1405919250032.60522dfa3cd1b4a2963c3a5aad56f06b.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:02:59,265 INFO
 [PostOpenDeployTasks:60522dfa3cd1b4a2963c3a5aad56f06b]
regionserver.HRegionServer: Finished post open deploy task for
image,tr\x00\x00\x00\x00\x04\xB1f\x0C,1405919250032.60522dfa3cd1b4a2963c3a5aad56f06b.
2014-08-22 15:03:00,231 INFO  [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Moving
r3c3s4.colo.vm,60020,1408711455393's hlogs to my queue
2014-08-22 15:03:00,245 INFO  [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Won't transfer the queue, another RS
took care of it because of: KeeperErrorCode = NoNode for
/hbase/replication/rs/r3c3s4.colo.vm,60020,1408711455393/lock
2014-08-22 15:03:34,296 INFO  [regionserver60020-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r3c5s3.colo.vm,60020,1408711481735 znode expired, triggering
replicatorRemoved event
2014-08-22 15:03:34,823 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
2014-08-22 15:03:34,851 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x021\xCD\xDBh,1399930900713.246d9c429f38cd6c87ed1c15541be1c3.
2014-08-22 15:03:34,881 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
2014-08-22 15:03:34,911 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,dk\x00\x00\x00\x00v5\x0E;,1406781032737.bb89220cb394a0e6e5d126513144bb1d.
2014-08-22 15:03:34,926 INFO
 [StoreOpener-ab9367ef6041f7a756117b5e22b7d566-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:34,926 INFO
 [StoreOpener-246d9c429f38cd6c87ed1c15541be1c3-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:34,936 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 246d9c429f38cd6c87ed1c15541be1c3; next
sequenceid=797536166
2014-08-22 15:03:34,941 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fi\x00\x00\x00\x00\x0C\xAF\xB4u,1405990240037.d7d06b01882d3965f42f81ccfd6709da.
2014-08-22 15:03:34,943 INFO  [StoreFileOpenerThread-cf_feature_vector-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 9ed6c933753e462698acc816b6b66477
2014-08-22 15:03:34,951 INFO
 [PostOpenDeployTasks:246d9c429f38cd6c87ed1c15541be1c3]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x021\xCD\xDBh,1399930900713.246d9c429f38cd6c87ed1c15541be1c3.
2014-08-22 15:03:34,953 INFO
 [StoreOpener-3b16a20aafe6c66168fdfb4b9e71c7da-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:34,956 INFO  [StoreFileOpenerThread-cf_feature_vector-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for dfe815de2af646539b8f51fdbbe4b929
2014-08-22 15:03:34,958 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined ab9367ef6041f7a756117b5e22b7d566; next
sequenceid=857629192
2014-08-22 15:03:34,964 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for deb5fa28c5de4e859614b014ff8f52a4
2014-08-22 15:03:34,966 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 3b16a20aafe6c66168fdfb4b9e71c7da; next
sequenceid=821191177
2014-08-22 15:03:34,971 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\x9B\x7F\xC95,1402431337599.969547bff4ba635df24f3597a829ec78.
2014-08-22 15:03:34,981 INFO
 [PostOpenDeployTasks:ab9367ef6041f7a756117b5e22b7d566]
regionserver.HRegionServer: Post open deploy tasks for
region=item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
2014-08-22 15:03:34,981 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
2014-08-22 15:03:34,981 INFO
 [PostOpenDeployTasks:3b16a20aafe6c66168fdfb4b9e71c7da]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
2014-08-22 15:03:34,983 INFO
 [PostOpenDeployTasks:246d9c429f38cd6c87ed1c15541be1c3] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x021\xCD\xDBh,1399930900713.246d9c429f38cd6c87ed1c15541be1c3.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:34,983 INFO
 [PostOpenDeployTasks:246d9c429f38cd6c87ed1c15541be1c3]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x021\xCD\xDBh,1399930900713.246d9c429f38cd6c87ed1c15541be1c3.
2014-08-22 15:03:35,001 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
2014-08-22 15:03:35,003 INFO
 [PostOpenDeployTasks:ab9367ef6041f7a756117b5e22b7d566] catalog.MetaEditor:
Updated row
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,003 INFO
 [PostOpenDeployTasks:ab9367ef6041f7a756117b5e22b7d566]
regionserver.HRegionServer: Finished post open deploy task for
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
2014-08-22 15:03:35,006 INFO
 [PostOpenDeployTasks:3b16a20aafe6c66168fdfb4b9e71c7da] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,006 INFO
 [PostOpenDeployTasks:3b16a20aafe6c66168fdfb4b9e71c7da]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
2014-08-22 15:03:35,044 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 15:03:35,085 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Open
image,se\x00\x00\x00\x00,\xAA\xCD,,1407172090506.efb6f85fa856f07db6284d6aecde2c22.
2014-08-22 15:03:35,088 INFO
 [StoreOpener-bb89220cb394a0e6e5d126513144bb1d-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,119 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 86c9ef0fa2dc4ad488d1f0d47d3bbf1a
2014-08-22 15:03:35,121 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined bb89220cb394a0e6e5d126513144bb1d; next
sequenceid=826521768
2014-08-22 15:03:35,127 INFO
 [PostOpenDeployTasks:bb89220cb394a0e6e5d126513144bb1d]
regionserver.HRegionServer: Post open deploy tasks for
region=image,dk\x00\x00\x00\x00v5\x0E;,1406781032737.bb89220cb394a0e6e5d126513144bb1d.
2014-08-22 15:03:35,142 INFO
 [StoreOpener-d7d06b01882d3965f42f81ccfd6709da-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,152 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for fc3fbfe5d21649568ec29aae7399bd4c
2014-08-22 15:03:35,154 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined d7d06b01882d3965f42f81ccfd6709da; next
sequenceid=826462229
2014-08-22 15:03:35,170 INFO
 [PostOpenDeployTasks:bb89220cb394a0e6e5d126513144bb1d] catalog.MetaEditor:
Updated row
image,dk\x00\x00\x00\x00v5\x0E;,1406781032737.bb89220cb394a0e6e5d126513144bb1d.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,170 INFO
 [PostOpenDeployTasks:bb89220cb394a0e6e5d126513144bb1d]
regionserver.HRegionServer: Finished post open deploy task for
image,dk\x00\x00\x00\x00v5\x0E;,1406781032737.bb89220cb394a0e6e5d126513144bb1d.
2014-08-22 15:03:35,171 INFO
 [StoreOpener-969547bff4ba635df24f3597a829ec78-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,177 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 969547bff4ba635df24f3597a829ec78; next
sequenceid=825161857
2014-08-22 15:03:35,180 INFO
 [PostOpenDeployTasks:d7d06b01882d3965f42f81ccfd6709da]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fi\x00\x00\x00\x00\x0C\xAF\xB4u,1405990240037.d7d06b01882d3965f42f81ccfd6709da.
2014-08-22 15:03:35,226 INFO
 [PostOpenDeployTasks:969547bff4ba635df24f3597a829ec78]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\x9B\x7F\xC95,1402431337599.969547bff4ba635df24f3597a829ec78.
2014-08-22 15:03:35,240 INFO
 [PostOpenDeployTasks:d7d06b01882d3965f42f81ccfd6709da] catalog.MetaEditor:
Updated row
image,fi\x00\x00\x00\x00\x0C\xAF\xB4u,1405990240037.d7d06b01882d3965f42f81ccfd6709da.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,240 INFO
 [PostOpenDeployTasks:d7d06b01882d3965f42f81ccfd6709da]
regionserver.HRegionServer: Finished post open deploy task for
image,fi\x00\x00\x00\x00\x0C\xAF\xB4u,1405990240037.d7d06b01882d3965f42f81ccfd6709da.
2014-08-22 15:03:35,319 INFO
 [PostOpenDeployTasks:969547bff4ba635df24f3597a829ec78] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\x9B\x7F\xC95,1402431337599.969547bff4ba635df24f3597a829ec78.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,319 INFO
 [PostOpenDeployTasks:969547bff4ba635df24f3597a829ec78]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\x9B\x7F\xC95,1402431337599.969547bff4ba635df24f3597a829ec78.
2014-08-22 15:03:35,413 INFO
 [StoreOpener-eb0c6870c459ffe5d21ff223b9169b56-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,421 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined eb0c6870c459ffe5d21ff223b9169b56; next
sequenceid=793193777
2014-08-22 15:03:35,448 INFO
 [PostOpenDeployTasks:eb0c6870c459ffe5d21ff223b9169b56]
regionserver.HRegionServer: Post open deploy tasks for
region=image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
2014-08-22 15:03:35,451 INFO
 [StoreOpener-50f9d7e3b62898fcea5b306b59474db6-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,461 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 60256fe2d4c248aebdbbbe176924cbdb
2014-08-22 15:03:35,462 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 50f9d7e3b62898fcea5b306b59474db6; next
sequenceid=848082246
2014-08-22 15:03:35,476 INFO
 [StoreOpener-23ba7e5b9354d05a71c526d2fa0ecd47-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,478 INFO
 [PostOpenDeployTasks:50f9d7e3b62898fcea5b306b59474db6]
regionserver.HRegionServer: Post open deploy tasks for
region=image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
2014-08-22 15:03:35,480 INFO
 [PostOpenDeployTasks:eb0c6870c459ffe5d21ff223b9169b56] catalog.MetaEditor:
Updated row
image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,480 INFO
 [PostOpenDeployTasks:eb0c6870c459ffe5d21ff223b9169b56]
regionserver.HRegionServer: Finished post open deploy task for
image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
2014-08-22 15:03:35,489 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 23ba7e5b9354d05a71c526d2fa0ecd47; next
sequenceid=755730346
2014-08-22 15:03:35,510 INFO
 [PostOpenDeployTasks:23ba7e5b9354d05a71c526d2fa0ecd47]
regionserver.HRegionServer: Post open deploy tasks for
region=image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 15:03:35,512 INFO
 [PostOpenDeployTasks:50f9d7e3b62898fcea5b306b59474db6] catalog.MetaEditor:
Updated row
image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,512 INFO
 [PostOpenDeployTasks:50f9d7e3b62898fcea5b306b59474db6]
regionserver.HRegionServer: Finished post open deploy task for
image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
2014-08-22 15:03:35,524 INFO
 [PostOpenDeployTasks:23ba7e5b9354d05a71c526d2fa0ecd47] catalog.MetaEditor:
Updated row
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,524 INFO
 [PostOpenDeployTasks:23ba7e5b9354d05a71c526d2fa0ecd47]
regionserver.HRegionServer: Finished post open deploy task for
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 15:03:35,561 INFO
 [StoreOpener-efb6f85fa856f07db6284d6aecde2c22-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:03:35,566 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 63293fdc236d480daf03ff70bc34c174
2014-08-22 15:03:35,571 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 71760500775d4061825984efa6b0ab2d
2014-08-22 15:03:35,579 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined efb6f85fa856f07db6284d6aecde2c22; next
sequenceid=844862826
2014-08-22 15:03:35,594 INFO
 [PostOpenDeployTasks:efb6f85fa856f07db6284d6aecde2c22]
regionserver.HRegionServer: Post open deploy tasks for
region=image,se\x00\x00\x00\x00,\xAA\xCD,,1407172090506.efb6f85fa856f07db6284d6aecde2c22.
2014-08-22 15:03:35,614 INFO
 [PostOpenDeployTasks:efb6f85fa856f07db6284d6aecde2c22] catalog.MetaEditor:
Updated row
image,se\x00\x00\x00\x00,\xAA\xCD,,1407172090506.efb6f85fa856f07db6284d6aecde2c22.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:03:35,614 INFO
 [PostOpenDeployTasks:efb6f85fa856f07db6284d6aecde2c22]
regionserver.HRegionServer: Finished post open deploy task for
image,se\x00\x00\x00\x00,\xAA\xCD,,1407172090506.efb6f85fa856f07db6284d6aecde2c22.
2014-08-22 15:03:36,631 INFO  [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Moving
r3c5s3.colo.vm,60020,1408711481735's hlogs to my queue
2014-08-22 15:03:36,638 INFO  [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Won't transfer the queue, another RS
took care of it because of: KeeperErrorCode = NoNode for
/hbase/replication/rs/r3c5s3.colo.vm,60020,1408711481735/lock
2014-08-22 15:08:45,292 INFO  [regionserver60020.periodicFlusher]
regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
flush for region
item,it\x00\x00\x00\x00\x05\xCC<X,1408619395176.19560c73dc6038a658d94ee627eabb90.
after a delay of 5511
2014-08-22 15:08:50,848 INFO  [MemStoreFlusher.0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=827083295,
memsize=14.3 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/item/19560c73dc6038a658d94ee627eabb90/.tmp/65ae64d279cd43e7b8b05a597d5c9626
2014-08-22 15:08:50,866 INFO  [MemStoreFlusher.0] regionserver.HStore:
Added
hdfs://visualmeta/hbase/data/default/item/19560c73dc6038a658d94ee627eabb90/cf_feature_vector/65ae64d279cd43e7b8b05a597d5c9626,
entries=78, sequenceid=827083295, filesize=5.5 K
2014-08-22 15:08:50,866 INFO  [MemStoreFlusher.0] regionserver.HRegion:
Finished memstore flush of ~14.3 K/14664, currentsize=0/0 for region
item,it\x00\x00\x00\x00\x05\xCC<X,1408619395176.19560c73dc6038a658d94ee627eabb90.
in 62ms, sequenceid=827083295, compaction requested=true
2014-08-22 15:09:35,289 WARN
 [defaultRpcServer.handler=23,queue=2,port=60020] hdfs.BlockReaderFactory:
I/O error constructing remote block reader.
java.io.IOException: Got error for OP_READ_BLOCK, self=/192.168.3.179:56326,
remote=/192.168.3.63:50010, for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1293205473_253317737
at
org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:432)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:397)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:786)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:665)
at
org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:325)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1025)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:968)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1301)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:90)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1223)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1430)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1312)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:387)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:524)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:572)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:257)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:173)
at
org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:269)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:695)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:683)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:533)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3937)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4017)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3885)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3867)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3854)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4827)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4801)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2849)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29583)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)
2014-08-22 15:09:35,289 WARN
 [defaultRpcServer.handler=23,queue=2,port=60020] hdfs.DFSClient:
Connection failure: Failed to connect to /192.168.3.63:50010 for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb
for block
BP-1157637685-192.168.3.192-1382642140917:blk_1293205473_253317737:java.io.IOException:
Got error for OP_READ_BLOCK, self=/192.168.3.179:56326, remote=/
192.168.3.63:50010, for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1293205473_253317737
java.io.IOException: Got error for OP_READ_BLOCK, self=/192.168.3.179:56326,
remote=/192.168.3.63:50010, for file
/hbase/data/default/image/1db3051c8d41943892c0230cb75bc1f2/cf/75ae96ea8ca2477e8f83291d7e1fe7cb,
for pool BP-1157637685-192.168.3.192-1382642140917 block
1293205473_253317737
at
org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:432)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:397)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:786)
at
org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:665)
at
org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:325)
at
org.apache.hadoop.hdfs.DFSInputStream.actualGetFromOneDataNode(DFSInputStream.java:1025)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchBlockByteRange(DFSInputStream.java:968)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1301)
at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:90)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1223)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1430)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1312)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:387)
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:253)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:524)
at
org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.reseekTo(HFileReaderV2.java:572)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseekAtOrAfter(StoreFileScanner.java:257)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:173)
at
org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:313)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:269)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:695)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.seekAsDirection(StoreScanner.java:683)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:533)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:140)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.populateResult(HRegion.java:3937)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:4017)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegion.java:3885)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3867)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3854)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4827)
at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4801)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2849)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29583)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2026)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)
2014-08-22 15:11:55,830 INFO  [regionserver60020-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r3c3s3.colo.vm,60020,1408696671696 znode expired, triggering
replicatorRemoved event
2014-08-22 15:11:56,530 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open item,de\x00\x00\x00\x00\x13\xD2G@
,1408558905273.aafeddc1719a1443b4106ff225b03f0c.
2014-08-22 15:11:56,555 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x00\xF7\x1F\xCF\x14,1395571953942.76d5202b21f739ed2cfd10f70bfb643f.
2014-08-22 15:11:56,585 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02_\xC8\x94\x92,1402344949643.0be3775016616493c42e2b16cc9318dd.
2014-08-22 15:11:56,621 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,de\x00\x00\x00\x02\x82\x8Dr\x10,1408181214855.3e5f0743b69c4a1f5b0e45dbdb4b82ee.
2014-08-22 15:11:56,644 INFO
 [StoreOpener-aafeddc1719a1443b4106ff225b03f0c-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,651 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,es\x00\x00\x00\x00%\xC1\xF7(,1407279563106.aacf1fbdad44db2c0438877a37ac78d5.
2014-08-22 15:11:56,655 INFO
 [StoreOpener-76d5202b21f739ed2cfd10f70bfb643f-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,661 INFO  [StoreFileOpenerThread-cf_feature_vector-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 501762388a0144ebb2a855fc635cdec3
2014-08-22 15:11:56,670 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined aafeddc1719a1443b4106ff225b03f0c; next
sequenceid=827838722
2014-08-22 15:11:56,676 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 052722f3641e4fad84f8f8b99f8b29e1
2014-08-22 15:11:56,687 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\x98\xC5\xD5\xD3,1401564947642.a8812708f97e06cb189c1a6df01473f5.
2014-08-22 15:11:56,688 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 76d5202b21f739ed2cfd10f70bfb643f; next
sequenceid=826511417
2014-08-22 15:11:56,691 INFO
 [StoreOpener-0be3775016616493c42e2b16cc9318dd-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,694 INFO
 [PostOpenDeployTasks:aafeddc1719a1443b4106ff225b03f0c]
regionserver.HRegionServer: Post open deploy tasks for
region=item,de\x00\x00\x00\x00\x13\xD2G@
,1408558905273.aafeddc1719a1443b4106ff225b03f0c.
2014-08-22 15:11:56,699 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 2d53061897524242a7db969444860ee9
2014-08-22 15:11:56,706 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for aa3ab7c419d940a48e5f221169a21298
2014-08-22 15:11:56,716 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,fr\x00\x00\x00\x00\xA8\xD8\xB7N,1407180602813.44f0f9f806475a2a68d20a4741b2d627.
2014-08-22 15:11:56,716 INFO
 [PostOpenDeployTasks:76d5202b21f739ed2cfd10f70bfb643f]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x00\xF7\x1F\xCF\x14,1395571953942.76d5202b21f739ed2cfd10f70bfb643f.
2014-08-22 15:11:56,716 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 0be3775016616493c42e2b16cc9318dd; next
sequenceid=826424041
2014-08-22 15:11:56,726 INFO
 [PostOpenDeployTasks:aafeddc1719a1443b4106ff225b03f0c] catalog.MetaEditor:
Updated row item,de\x00\x00\x00\x00\x13\xD2G@,1408558905273.aafeddc1719a1443b4106ff225b03f0c.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,727 INFO
 [PostOpenDeployTasks:aafeddc1719a1443b4106ff225b03f0c]
regionserver.HRegionServer: Finished post open deploy task for
item,de\x00\x00\x00\x00\x13\xD2G@
,1408558905273.aafeddc1719a1443b4106ff225b03f0c.
2014-08-22 15:11:56,760 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,it\x00\x00\x00\x006\x04\xBD;,1401330620861.cbea316034f5a456e4f7866a988e56fb.
2014-08-22 15:11:56,760 INFO
 [PostOpenDeployTasks:0be3775016616493c42e2b16cc9318dd]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02_\xC8\x94\x92,1402344949643.0be3775016616493c42e2b16cc9318dd.
2014-08-22 15:11:56,762 INFO
 [PostOpenDeployTasks:76d5202b21f739ed2cfd10f70bfb643f] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x00\xF7\x1F\xCF\x14,1395571953942.76d5202b21f739ed2cfd10f70bfb643f.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,762 INFO
 [PostOpenDeployTasks:76d5202b21f739ed2cfd10f70bfb643f]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x00\xF7\x1F\xCF\x14,1395571953942.76d5202b21f739ed2cfd10f70bfb643f.
2014-08-22 15:11:56,788 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,nl\x00\x00\x00\x002\xE1\xE6.,1405002295327.4bba527b4d574c7090e425e0f8f01969.
2014-08-22 15:11:56,790 INFO
 [PostOpenDeployTasks:0be3775016616493c42e2b16cc9318dd] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02_\xC8\x94\x92,1402344949643.0be3775016616493c42e2b16cc9318dd.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,790 INFO
 [PostOpenDeployTasks:0be3775016616493c42e2b16cc9318dd]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02_\xC8\x94\x92,1402344949643.0be3775016616493c42e2b16cc9318dd.
2014-08-22 15:11:56,826 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,se\x00\x00\x00\x00)S~a,1401960422149.cdb390aa949c463b320d5835a5eb29da.
2014-08-22 15:11:56,860 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Open
image,tr\x00\x00\x00\x00\x04T\x84',1405349548090.61a54649329084e73b43c331caf1f403.
2014-08-22 15:11:56,871 INFO
 [StoreOpener-3e5f0743b69c4a1f5b0e45dbdb4b82ee-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,882 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 3e5f0743b69c4a1f5b0e45dbdb4b82ee; next
sequenceid=858767012
2014-08-22 15:11:56,887 INFO
 [StoreOpener-aacf1fbdad44db2c0438877a37ac78d5-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,892 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 327e010537294dd08fabf648116cdfee
2014-08-22 15:11:56,893 INFO
 [PostOpenDeployTasks:3e5f0743b69c4a1f5b0e45dbdb4b82ee]
regionserver.HRegionServer: Post open deploy tasks for
region=image,de\x00\x00\x00\x02\x82\x8Dr\x10,1408181214855.3e5f0743b69c4a1f5b0e45dbdb4b82ee.
2014-08-22 15:11:56,897 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for b4339e4a561945eb845952109bf29e94
2014-08-22 15:11:56,904 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined aacf1fbdad44db2c0438877a37ac78d5; next
sequenceid=800476243
2014-08-22 15:11:56,926 INFO
 [PostOpenDeployTasks:aacf1fbdad44db2c0438877a37ac78d5]
regionserver.HRegionServer: Post open deploy tasks for
region=image,es\x00\x00\x00\x00%\xC1\xF7(,1407279563106.aacf1fbdad44db2c0438877a37ac78d5.
2014-08-22 15:11:56,928 INFO
 [PostOpenDeployTasks:3e5f0743b69c4a1f5b0e45dbdb4b82ee] catalog.MetaEditor:
Updated row
image,de\x00\x00\x00\x02\x82\x8Dr\x10,1408181214855.3e5f0743b69c4a1f5b0e45dbdb4b82ee.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,928 INFO
 [PostOpenDeployTasks:3e5f0743b69c4a1f5b0e45dbdb4b82ee]
regionserver.HRegionServer: Finished post open deploy task for
image,de\x00\x00\x00\x02\x82\x8Dr\x10,1408181214855.3e5f0743b69c4a1f5b0e45dbdb4b82ee.
2014-08-22 15:11:56,956 INFO
 [StoreOpener-a8812708f97e06cb189c1a6df01473f5-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:56,963 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined a8812708f97e06cb189c1a6df01473f5; next
sequenceid=828266015
2014-08-22 15:11:56,967 INFO
 [PostOpenDeployTasks:aacf1fbdad44db2c0438877a37ac78d5] catalog.MetaEditor:
Updated row
image,es\x00\x00\x00\x00%\xC1\xF7(,1407279563106.aacf1fbdad44db2c0438877a37ac78d5.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,967 INFO
 [PostOpenDeployTasks:aacf1fbdad44db2c0438877a37ac78d5]
regionserver.HRegionServer: Finished post open deploy task for
image,es\x00\x00\x00\x00%\xC1\xF7(,1407279563106.aacf1fbdad44db2c0438877a37ac78d5.
2014-08-22 15:11:56,976 INFO
 [PostOpenDeployTasks:a8812708f97e06cb189c1a6df01473f5]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\x98\xC5\xD5\xD3,1401564947642.a8812708f97e06cb189c1a6df01473f5.
2014-08-22 15:11:56,997 INFO
 [PostOpenDeployTasks:a8812708f97e06cb189c1a6df01473f5] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\x98\xC5\xD5\xD3,1401564947642.a8812708f97e06cb189c1a6df01473f5.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:56,997 INFO
 [PostOpenDeployTasks:a8812708f97e06cb189c1a6df01473f5]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\x98\xC5\xD5\xD3,1401564947642.a8812708f97e06cb189c1a6df01473f5.
2014-08-22 15:11:57,028 INFO
 [StoreOpener-44f0f9f806475a2a68d20a4741b2d627-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,033 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 269ec65f620947e1b65f16e5bbb7893a
2014-08-22 15:11:57,036 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for be8bdc230ac74f1c8dcd28c17882a0f9
2014-08-22 15:11:57,043 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined 44f0f9f806475a2a68d20a4741b2d627; next
sequenceid=830213588
2014-08-22 15:11:57,058 INFO
 [StoreOpener-cbea316034f5a456e4f7866a988e56fb-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,066 INFO
 [PostOpenDeployTasks:44f0f9f806475a2a68d20a4741b2d627]
regionserver.HRegionServer: Post open deploy tasks for
region=image,fr\x00\x00\x00\x00\xA8\xD8\xB7N,1407180602813.44f0f9f806475a2a68d20a4741b2d627.
2014-08-22 15:11:57,067 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 6bde84cd3f1a420ab5c615e52adb7ec4
2014-08-22 15:11:57,075 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for a433503e47814feb95f63467cd112d37
2014-08-22 15:11:57,077 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined cbea316034f5a456e4f7866a988e56fb; next
sequenceid=826598203
2014-08-22 15:11:57,089 INFO
 [PostOpenDeployTasks:cbea316034f5a456e4f7866a988e56fb]
regionserver.HRegionServer: Post open deploy tasks for
region=image,it\x00\x00\x00\x006\x04\xBD;,1401330620861.cbea316034f5a456e4f7866a988e56fb.
2014-08-22 15:11:57,091 INFO
 [PostOpenDeployTasks:44f0f9f806475a2a68d20a4741b2d627] catalog.MetaEditor:
Updated row
image,fr\x00\x00\x00\x00\xA8\xD8\xB7N,1407180602813.44f0f9f806475a2a68d20a4741b2d627.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,091 INFO
 [PostOpenDeployTasks:44f0f9f806475a2a68d20a4741b2d627]
regionserver.HRegionServer: Finished post open deploy task for
image,fr\x00\x00\x00\x00\xA8\xD8\xB7N,1407180602813.44f0f9f806475a2a68d20a4741b2d627.
2014-08-22 15:11:57,092 INFO
 [StoreOpener-4bba527b4d574c7090e425e0f8f01969-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,102 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 0f316ba775194f5f9235ed88cd2bc5d8
2014-08-22 15:11:57,105 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for a95360a0f3e348d99fe1b7a727fbe825
2014-08-22 15:11:57,106 INFO  [RS_OPEN_REGION-r3c7s4:60020-0]
regionserver.HRegion: Onlined 4bba527b4d574c7090e425e0f8f01969; next
sequenceid=821172061
2014-08-22 15:11:57,116 INFO
 [PostOpenDeployTasks:4bba527b4d574c7090e425e0f8f01969]
regionserver.HRegionServer: Post open deploy tasks for
region=image,nl\x00\x00\x00\x002\xE1\xE6.,1405002295327.4bba527b4d574c7090e425e0f8f01969.
2014-08-22 15:11:57,118 INFO
 [PostOpenDeployTasks:cbea316034f5a456e4f7866a988e56fb] catalog.MetaEditor:
Updated row
image,it\x00\x00\x00\x006\x04\xBD;,1401330620861.cbea316034f5a456e4f7866a988e56fb.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,118 INFO
 [PostOpenDeployTasks:cbea316034f5a456e4f7866a988e56fb]
regionserver.HRegionServer: Finished post open deploy task for
image,it\x00\x00\x00\x006\x04\xBD;,1401330620861.cbea316034f5a456e4f7866a988e56fb.
2014-08-22 15:11:57,134 INFO
 [PostOpenDeployTasks:4bba527b4d574c7090e425e0f8f01969] catalog.MetaEditor:
Updated row
image,nl\x00\x00\x00\x002\xE1\xE6.,1405002295327.4bba527b4d574c7090e425e0f8f01969.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,134 INFO
 [PostOpenDeployTasks:4bba527b4d574c7090e425e0f8f01969]
regionserver.HRegionServer: Finished post open deploy task for
image,nl\x00\x00\x00\x002\xE1\xE6.,1405002295327.4bba527b4d574c7090e425e0f8f01969.
2014-08-22 15:11:57,147 INFO
 [StoreOpener-cdb390aa949c463b320d5835a5eb29da-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,168 INFO  [RS_OPEN_REGION-r3c7s4:60020-2]
regionserver.HRegion: Onlined cdb390aa949c463b320d5835a5eb29da; next
sequenceid=826595433
2014-08-22 15:11:57,182 INFO
 [StoreOpener-61a54649329084e73b43c331caf1f403-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; delete expired; major period 604800000, major jitter 0.500000
2014-08-22 15:11:57,185 INFO
 [PostOpenDeployTasks:cdb390aa949c463b320d5835a5eb29da]
regionserver.HRegionServer: Post open deploy tasks for
region=image,se\x00\x00\x00\x00)S~a,1401960422149.cdb390aa949c463b320d5835a5eb29da.
2014-08-22 15:11:57,197 INFO  [StoreFileOpenerThread-cf-1]
regionserver.StoreFile$Reader: Loaded Delete Family Bloom
(CompoundBloomFilter) metadata for 91c192751cf346ae91678bee7b7d86d5
2014-08-22 15:11:57,198 INFO  [RS_OPEN_REGION-r3c7s4:60020-1]
regionserver.HRegion: Onlined 61a54649329084e73b43c331caf1f403; next
sequenceid=795164401
2014-08-22 15:11:57,199 INFO
 [PostOpenDeployTasks:cdb390aa949c463b320d5835a5eb29da] catalog.MetaEditor:
Updated row
image,se\x00\x00\x00\x00)S~a,1401960422149.cdb390aa949c463b320d5835a5eb29da.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,199 INFO
 [PostOpenDeployTasks:cdb390aa949c463b320d5835a5eb29da]
regionserver.HRegionServer: Finished post open deploy task for
image,se\x00\x00\x00\x00)S~a,1401960422149.cdb390aa949c463b320d5835a5eb29da.
2014-08-22 15:11:57,218 INFO
 [PostOpenDeployTasks:61a54649329084e73b43c331caf1f403]
regionserver.HRegionServer: Post open deploy tasks for
region=image,tr\x00\x00\x00\x00\x04T\x84',1405349548090.61a54649329084e73b43c331caf1f403.
2014-08-22 15:11:57,247 INFO
 [PostOpenDeployTasks:61a54649329084e73b43c331caf1f403] catalog.MetaEditor:
Updated row
image,tr\x00\x00\x00\x00\x04T\x84',1405349548090.61a54649329084e73b43c331caf1f403.
with server=r3c7s4.colo.vm,60020,1408707352291
2014-08-22 15:11:57,247 INFO
 [PostOpenDeployTasks:61a54649329084e73b43c331caf1f403]
regionserver.HRegionServer: Finished post open deploy task for
image,tr\x00\x00\x00\x00\x04T\x84',1405349548090.61a54649329084e73b43c331caf1f403.
2014-08-22 15:11:57,950 INFO  [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Moving
r3c3s3.colo.vm,60020,1408696671696's hlogs to my queue
2014-08-22 15:11:57,958 INFO  [ReplicationExecutor-0]
replication.ReplicationQueuesZKImpl: Won't transfer the queue, another RS
took care of it because of: KeeperErrorCode = NoNode for
/hbase/replication/rs/r3c3s3.colo.vm,60020,1408696671696/lock
2014-08-22 15:12:39,348 INFO
 [PriorityRpcServer.handler=8,queue=0,port=60020]
regionserver.HRegionServer: Close ab9367ef6041f7a756117b5e22b7d566, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:39,445 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=857629207,
memsize=5.1 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/item/ab9367ef6041f7a756117b5e22b7d566/.tmp/eb964aa729f94df5b3f55d7b9c56f5e2
2014-08-22 15:12:39,472 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HStore: Added
hdfs://visualmeta/hbase/data/default/item/ab9367ef6041f7a756117b5e22b7d566/cf_feature_vector/eb964aa729f94df5b3f55d7b9c56f5e2,
entries=28, sequenceid=857629207, filesize=2.6 K
2014-08-22 15:12:39,472 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Finished memstore flush of ~5.1 K/5264,
currentsize=0/0 for region
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
in 111ms, sequenceid=857629207, compaction requested=true
2014-08-22 15:12:39,517 INFO
 [StoreCloserThread-item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.-1]
regionserver.HStore: Closed cf_feature_vector
2014-08-22 15:12:39,518 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
item,de\x00\x00\x00\x00\x0EY\x01(,1408563380665.ab9367ef6041f7a756117b5e22b7d566.
2014-08-22 15:12:39,518 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
ab9367ef6041f7a756117b5e22b7d566 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 857629192
2014-08-22 15:12:39,698 INFO
 [PriorityRpcServer.handler=0,queue=0,port=60020]
regionserver.HRegionServer: Close 3129db940c9f94729b986b742f0161c2, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:39,710 INFO
 [PriorityRpcServer.handler=2,queue=0,port=60020]
regionserver.HRegionServer: Close dc121bab8ed97c25f4f135c4f4f6fdcc, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:39,824 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=830369747,
memsize=111.6 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/classifier/3129db940c9f94729b986b742f0161c2/.tmp/1fb37a3ae1e346bea58b4ff1babc6b88
2014-08-22 15:12:39,896 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.DefaultStoreFlusher: Flushed, sequenceid=827557339,
memsize=451.1 K, hasBloomFilter=true, into tmp file
hdfs://visualmeta/hbase/data/default/classifier/dc121bab8ed97c25f4f135c4f4f6fdcc/.tmp/377065ea713742e6b0dbc818499a787a
2014-08-22 15:12:39,899 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HStore: Added
hdfs://visualmeta/hbase/data/default/classifier/3129db940c9f94729b986b742f0161c2/cf/1fb37a3ae1e346bea58b4ff1babc6b88,
entries=5, sequenceid=830369747, filesize=112.0 K
2014-08-22 15:12:39,899 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Finished memstore flush of ~111.6 K/114240,
currentsize=0/0 for region
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
in 196ms, sequenceid=830369747, compaction requested=true
2014-08-22 15:12:39,923 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HStore: Added
hdfs://visualmeta/hbase/data/default/classifier/dc121bab8ed97c25f4f135c4f4f6fdcc/cf/377065ea713742e6b0dbc818499a787a,
entries=8, sequenceid=827557339, filesize=451.4 K
2014-08-22 15:12:39,923 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Finished memstore flush of ~451.1 K/461928,
currentsize=0/0 for region
classifier,\xB3\x87H,1408395375533.dc121bab8ed97c25f4f135c4f4f6fdcc. in
213ms, sequenceid=827557339, compaction requested=true
2014-08-22 15:12:39,947 INFO
 [StoreCloserThread-classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:39,947 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
classifier,\xE9\x02-\xF0\xA2\xDF\x17*,1408110936897.3129db940c9f94729b986b742f0161c2.
2014-08-22 15:12:39,948 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
3129db940c9f94729b986b742f0161c2 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 830369735
2014-08-22 15:12:39,965 INFO
 [StoreCloserThread-classifier,\xB3\x87H,1408395375533.dc121bab8ed97c25f4f135c4f4f6fdcc.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:39,965 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
classifier,\xB3\x87H,1408395375533.dc121bab8ed97c25f4f135c4f4f6fdcc.
2014-08-22 15:12:39,966 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
dc121bab8ed97c25f4f135c4f4f6fdcc to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 827557330
2014-08-22 15:12:43,210 INFO
 [PriorityRpcServer.handler=5,queue=0,port=60020]
regionserver.HRegionServer: Close de56dc1dc39aa245d466bb86158f0709, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,235 INFO
 [PriorityRpcServer.handler=4,queue=0,port=60020]
regionserver.HRegionServer: Close d26e27c320268993613c8c0c4ef6eb57, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,245 INFO
 [StoreCloserThread-image,fr\x00\x00\x00\x00\x98\xEAI\x8A,1401591381273.de56dc1dc39aa245d466bb86158f0709.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,245 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,fr\x00\x00\x00\x00\x98\xEAI\x8A,1401591381273.de56dc1dc39aa245d466bb86158f0709.
2014-08-22 15:12:43,245 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
de56dc1dc39aa245d466bb86158f0709 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 855087042
2014-08-22 15:12:43,258 INFO
 [PriorityRpcServer.handler=6,queue=0,port=60020]
regionserver.HRegionServer: Close 688b0564f44b3dbf080a1ef421374261, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,261 INFO
 [StoreCloserThread-image,se\x00\x00\x00\x00)\xF5\xC3F,1402967065165.d26e27c320268993613c8c0c4ef6eb57.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,262 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,se\x00\x00\x00\x00)\xF5\xC3F,1402967065165.d26e27c320268993613c8c0c4ef6eb57.
2014-08-22 15:12:43,262 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
d26e27c320268993613c8c0c4ef6eb57 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 855084513
2014-08-22 15:12:43,270 INFO
 [PriorityRpcServer.handler=7,queue=0,port=60020]
regionserver.HRegionServer: Close f266cc9eb0de018f335dab7592b08db4, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,278 INFO  [StoreCloserThread-image,es\x00\x00\x00\x00
\xA2tG,1400138220753.688b0564f44b3dbf080a1ef421374261.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,279 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed image,es\x00\x00\x00\x00
\xA2tG,1400138220753.688b0564f44b3dbf080a1ef421374261.
2014-08-22 15:12:43,279 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
688b0564f44b3dbf080a1ef421374261 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 756505811
2014-08-22 15:12:43,294 INFO
 [PriorityRpcServer.handler=3,queue=0,port=60020]
regionserver.HRegionServer: Close 50f9d7e3b62898fcea5b306b59474db6, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,300 INFO
 [StoreCloserThread-image,fi\x00\x00\x00\x00\x0D\x96\xF3\xD1,1408470461149.f266cc9eb0de018f335dab7592b08db4.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,300 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,fi\x00\x00\x00\x00\x0D\x96\xF3\xD1,1408470461149.f266cc9eb0de018f335dab7592b08db4.
2014-08-22 15:12:43,300 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
f266cc9eb0de018f335dab7592b08db4 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 846307853
2014-08-22 15:12:43,329 INFO
 [PriorityRpcServer.handler=9,queue=0,port=60020]
regionserver.HRegionServer: Close eb0c6870c459ffe5d21ff223b9169b56, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,337 INFO
 [StoreCloserThread-image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,337 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,it\x00\x00\x00\x00<\xC9H\xE7,1406031094658.50f9d7e3b62898fcea5b306b59474db6.
2014-08-22 15:12:43,337 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
50f9d7e3b62898fcea5b306b59474db6 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 848082246
2014-08-22 15:12:43,345 INFO
 [StoreCloserThread-image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,346 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,in\x00\x00\x00\x00\x01F\xE73,1400549232558.eb0c6870c459ffe5d21ff223b9169b56.
2014-08-22 15:12:43,346 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
eb0c6870c459ffe5d21ff223b9169b56 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 793193777
2014-08-22 15:12:43,348 INFO
 [PriorityRpcServer.handler=1,queue=0,port=60020]
regionserver.HRegionServer: Close d65a237e26153d29201b57dd4870c590, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,360 INFO
 [PriorityRpcServer.handler=8,queue=0,port=60020]
regionserver.HRegionServer: Close 492dc247123bfea48058d447e5a3d7ab, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,372 INFO
 [PriorityRpcServer.handler=0,queue=0,port=60020]
regionserver.HRegionServer: Close 23ba7e5b9354d05a71c526d2fa0ecd47, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,376 INFO
 [StoreCloserThread-image,in\x00\x00\x00\x00\x01\x04WI,1399325596246.492dc247123bfea48058d447e5a3d7ab.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,377 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,in\x00\x00\x00\x00\x01\x04WI,1399325596246.492dc247123bfea48058d447e5a3d7ab.
2014-08-22 15:12:43,377 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
492dc247123bfea48058d447e5a3d7ab to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 691695328
2014-08-22 15:12:43,378 INFO
 [PriorityRpcServer.handler=2,queue=0,port=60020]
regionserver.HRegionServer: Close 3b16a20aafe6c66168fdfb4b9e71c7da, via
zk=yes, znode version=0, on r3c5s3.colo.vm,60020,1408712894162
2014-08-22 15:12:43,381 INFO
 [StoreCloserThread-image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,381 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegion: Closed
image,ru\x00\x00\x00\x00\x00\xF9\xB9\x9C,1400121006136.23ba7e5b9354d05a71c526d2fa0ecd47.
2014-08-22 15:12:43,381 INFO  [RS_CLOSE_REGION-r3c7s4:60020-0]
regionserver.HRegionServer: Adding moved region record:
23ba7e5b9354d05a71c526d2fa0ecd47 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 755730346
2014-08-22 15:12:43,388 INFO
 [StoreCloserThread-image,fr\x00\x00\x00\x00\x9B+\x9A\xA7,1402266525879.d65a237e26153d29201b57dd4870c590.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,388 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegion: Closed
image,fr\x00\x00\x00\x00\x9B+\x9A\xA7,1402266525879.d65a237e26153d29201b57dd4870c590.
2014-08-22 15:12:43,388 INFO  [RS_CLOSE_REGION-r3c7s4:60020-1]
regionserver.HRegionServer: Adding moved region record:
d65a237e26153d29201b57dd4870c590 to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 824275348
2014-08-22 15:12:43,421 INFO
 [StoreCloserThread-image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.-1]
regionserver.HStore: Closed cf
2014-08-22 15:12:43,422 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegion: Closed
image,de\x00\x00\x00\x02u\xF7\x92\x10,1405380638297.3b16a20aafe6c66168fdfb4b9e71c7da.
2014-08-22 15:12:43,422 INFO  [RS_CLOSE_REGION-r3c7s4:60020-2]
regionserver.HRegionServer: Adding moved region record:
3b16a20aafe6c66168fdfb4b9e71c7da to
r3c5s3.colo.vm,60020,1408712894162:60020 as of 821191177


RS Queue:
===========================================================
Compaction/Split Queue summary: compaction_queue=(0:0), split_queue=0,
merge_queue=0
Compaction/Split Queue dump:
  LargeCompation Queue:

  SmallCompation Queue:

  Split Queue:

  Region Merge Queue:


Flush Queue summary: flush_queue=0
Flush Queue Queue dump:
  Flush Queue:




On Fri, Aug 22, 2014 at 7:38 PM, Stack <stack@duboce.net> wrote:

> Do you have a few thread dumps from a 'deaf' instance?
> St.Ack
>
>
>
> On Fri, Aug 22, 2014 at 10:28 AM, Johannes Schaback <
> johannes.schaback@visual-meta.com> wrote:
>
> > Dear HBase-Pros,
> >
> > we face a serious issue with our HBase production cluster for two days
> now.
> > Every couple minutes, a random RegionServer gets stuck and does not
> process
> > any requests. In addition this causes the other RegionServers to
> > freeze within a minute which brings down the entire cluster. Stopping the
> > affected RegionServer unblocks the cluster and everything comes back to
> > normal.
> >
> > We run 27 RegionServers, each having 31 GB JVM memory. The HBase Version
> is
> > 0.98.5 on Hadoop 2.4.1. We basically have two tables, the first having
> > about 4,500 Regions and holding 8 TB with 1000 requests per second, the
> > second table is around 200 Regions with about 50,000 to 120,000 requests
> > per sec over all Regions, 800 GB worth of data and with IN_MEMORY
> enabled.
> >
> > While investigating the problem, I found out, that every healthy
> > RegionServer has the following thread:
> >
> > Thread 12 (RpcServer.listener,port=60020):
> >   State: RUNNABLE
> >   Blocked count: 35
> >   Waited count: 0
> >   Stack:
> >     sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> >     sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
> >     sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
> >     sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
> >     sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
> >
>  org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:684)
> >
> >
> > When suddenly becoming a blocked RegionServer, this particular thread
> then
> > looks like
> >
> > Thread 12 (RpcServer.listener,port=60020):
> >   State: BLOCKED
> >   Blocked count: 2889
> >   Waited count: 0
> >   Blocked on
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@38cba1a1
> >   Blocked by 14 (RpcServer.reader=1,port=60020)
> >   Stack:
> >
> >
> >
> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.registerChannel(RpcServer.java:619)
> >
> >
> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept(RpcServer.java:774)
> >
>  org.apache.hadoop.hbase.ipc.RpcServer$Listener.run(RpcServer.java:692)
> >
> >
> > Also, JMX shows for an unhealthy RegionServer that
> >
> >    - "queueSize" grows quickly and constantly to values greater than 60k,
> >    and
> >    - "numCallsInGeneralQueue" quickly reaches 300
> >
> > Both values are usually very small or 0 under normal circumstances, but
> in
> > case of a RS "getting stuck" they explode, which leads me to believe that
> > the IPC-queue does not get processed properly causing the RegionServer to
> > become "deaf".
> >
> > These two symptoms appear to bring down the entire cluster. When killign
> > that RS, everyhing goes back to normal.
> >
> > I could not find any correlation between this phenomenon and compactions,
> > load or other factors. hbck says it is all fine as well.
> >
> > The servers are all 3.2.0-4-amd64 Debian, 12 cores, 96 GB RAM. Besides
> the
> > RS and a DataNode, there isn't too much running on the boxes so the load
> > (top) is usually around 5 to 10 and bandwidth does not exceed 10 MB on
> > average.
> >
> > We currently survive by polling /jmx of all RegionServers constantly and
> > restarting those off that show the symptioms :(
> >
> > Do you have any idea what could be causing this?
> >
> > Thank you very much in advance!
> >
> > Johannes
> >
>



-- 
LadenZeile.de <http://www.ladenzeile.de/>
powered by Visual Meta GmbH - www.visual-meta.com

Tel.: +49 30 / 609 84 88 20
Fax: +49 30 / 609 84 88 21
E-Mail: johannes.schaback@visual-meta.com

Visual Meta GmbH, Schützenstraße 25, 10117 Berlin
Geschäftsführer: Robert M. Maier, Johannes Schaback
Handelsregister HRB 115795 B, Amtsgericht Charlottenburg
USt-IdNr.: DE263760203

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message