hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "zhangjg (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2
Date Tue, 14 Apr 2015 04:10:12 GMT
zhangjg created HBASE-13461:
-------------------------------

             Summary: RegionSever Hlog flush BLOCKED  on  hbase-0.96.2-hadoop2
                 Key: HBASE-13461
                 URL: https://issues.apache.org/jira/browse/HBASE-13461
             Project: HBase
          Issue Type: Bug
         Environment: hbase-0.96.2-hadoop2   hadoop2.2.0
            Reporter: zhangjg


I try to dump  thread stack below:

"RpcServer.handler=63,port=60020" daemon prio=10 tid=0x00007fdcddc5d000 nid=0x5f9 waiting
for monitor entry [0x00007fd289194000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
        - waiting to lock <0x00007fd36c023728> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        - locked <0x00007fd510cfdc28> (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream)
        at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833)
        at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843)
        at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075)
        - locked <0x00007fd2d9bbfad0> (a java.lang.Object)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240)
        at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185)
        at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889)

"RpcServer.handler=12,port=60020" daemon prio=10 tid=0x00007fdcddf2c800 nid=0x5c6 in Object.wait()
[0x00007fd28c4c7000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803)
        - locked <0x00007fd45857c540> (a java.util.LinkedList)
        at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697)
        at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590)
        at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575)
        at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240)
        at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185)
        at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889)

"RpcServer.handler=11,port=60020" daemon prio=10 tid=0x00007fdcdd9e1000 nid=0x5c5 in Object.wait()
[0x00007fd28c5c8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803)
        - locked <0x00007fd45857c540> (a java.util.LinkedList)
        at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697)
        at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590)
        at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575)
        at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240)
        at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185)
        at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889)


"RpcServer.handler=95,port=60020" daemon prio=10 tid=0x00007fdcdc50b800 nid=0x619 in Object.wait()
[0x00007fd287174000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:485)
        at org.apache.hadoop.hdfs.DFSOutputStream.waitAndQueueCurrentPacket(DFSOutputStream.java:1475)
        - locked <0x00007fd45857c540> (a java.util.LinkedList)
        at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1688)
        - locked <0x00007fd36c023728> (a org.apache.hadoop.hdfs.DFSOutputStream)
        at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590)
        at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575)
        at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121)
        at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240)
        at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185)
        at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889)




"DataStreamer for file /hbase/WALs/hdp910.qq.diditaxi.com,60020,1428079391214/hdp910.qq.diditaxi.com%2C60020%2C1428079391214.1428919313998
block BP-1892361854-10.231.149.77-1397112594861:blk_1320569732_10997627
53479" daemon prio=10 tid=0x00007fdccda8f800 nid=0x4c34 in Object.wait() [0x00007fd23646c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:491)
        - locked <0x00007fd45857c540> (a java.util.LinkedList)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message