Return-Path: X-Original-To: apmail-hbase-issues-archive@www.apache.org Delivered-To: apmail-hbase-issues-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 4DE6517813 for ; Tue, 14 Apr 2015 04:10:14 +0000 (UTC) Received: (qmail 11865 invoked by uid 500); 14 Apr 2015 04:10:13 -0000 Delivered-To: apmail-hbase-issues-archive@hbase.apache.org Received: (qmail 11297 invoked by uid 500); 14 Apr 2015 04:10:12 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 11073 invoked by uid 99); 14 Apr 2015 04:10:12 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Apr 2015 04:10:12 +0000 Date: Tue, 14 Apr 2015 04:10:12 +0000 (UTC) From: "zhangjg (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (HBASE-13461) RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 zhangjg created HBASE-13461: ------------------------------- Summary: RegionSever Hlog flush BLOCKED on hbase-0.96.2-hadoop2 Key: HBASE-13461 URL: https://issues.apache.org/jira/browse/HBASE-13461 Project: HBase Issue Type: Bug Environment: hbase-0.96.2-hadoop2 hadoop2.2.0 Reporter: zhangjg I try to dump thread stack below: "RpcServer.handler=63,port=60020" daemon prio=10 tid=0x00007fdcddc5d000 nid=0x5f9 waiting for monitor entry [0x00007fd289194000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98) - waiting to lock <0x00007fd36c023728> (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59) at java.io.DataOutputStream.write(DataOutputStream.java:90) - locked <0x00007fd510cfdc28> (a org.apache.hadoop.hdfs.client.HdfsDataOutputStream) at com.google.protobuf.CodedOutputStream.refreshBuffer(CodedOutputStream.java:833) at com.google.protobuf.CodedOutputStream.flush(CodedOutputStream.java:843) at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:91) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.append(ProtobufLogWriter.java:87) at org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.hlogFlush(FSHLog.java:1026) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1075) - locked <0x00007fd2d9bbfad0> (a java.lang.Object) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) "RpcServer.handler=12,port=60020" daemon prio=10 tid=0x00007fdcddf2c800 nid=0x5c6 in Object.wait() [0x00007fd28c4c7000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked <0x00007fd45857c540> (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) "RpcServer.handler=11,port=60020" daemon prio=10 tid=0x00007fdcdd9e1000 nid=0x5c5 in Object.wait() [0x00007fd28c5c8000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream.waitForAckedSeqno(DFSOutputStream.java:1803) - locked <0x00007fd45857c540> (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1697) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) "RpcServer.handler=95,port=60020" daemon prio=10 tid=0x00007fdcdc50b800 nid=0x619 in Object.wait() [0x00007fd287174000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at org.apache.hadoop.hdfs.DFSOutputStream.waitAndQueueCurrentPacket(DFSOutputStream.java:1475) - locked <0x00007fd45857c540> (a java.util.LinkedList) at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:1688) - locked <0x00007fd36c023728> (a org.apache.hadoop.hdfs.DFSOutputStream) at org.apache.hadoop.hdfs.DFSOutputStream.hflush(DFSOutputStream.java:1590) at org.apache.hadoop.hdfs.DFSOutputStream.sync(DFSOutputStream.java:1575) at org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:121) at org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter.sync(ProtobufLogWriter.java:135) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1098) at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1240) at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5593) at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2315) at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2028) at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4094) at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3380) at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3284) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:26935) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2185) at org.apache.hadoop.hbase.ipc.RpcServer$Handler.run(RpcServer.java:1889) "DataStreamer for file /hbase/WALs/hdp910.qq.diditaxi.com,60020,1428079391214/hdp910.qq.diditaxi.com%2C60020%2C1428079391214.1428919313998 block BP-1892361854-10.231.149.77-1397112594861:blk_1320569732_10997627 53479" daemon prio=10 tid=0x00007fdccda8f800 nid=0x4c34 in Object.wait() [0x00007fd23646c000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:491) - locked <0x00007fd45857c540> (a java.util.LinkedList) -- This message was sent by Atlassian JIRA (v6.3.4#6332)