hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "聪聪" <175998...@qq.com>
Subject 转发:in hdfs, part of the last contact value increases
Date Wed, 21 Dec 2016 09:07:51 GMT
------------------ 原始邮件 ------------------
发件人: "聪聪";<175998806@qq.com>;
发送时间: 2016年12月21日(星期三) 上午10:45
收件人: "user"<user@hbase.apache.org>; 

主题: in hdfs, part of the last contact value increases



Recently, I have increased two disks on a single datanode . I encountered a problem, in hdfs,
part of the last contact value increases, resulting in hbase the handler is occupied, hdfs
web ui specific as shown below.



The datanode prints the log as follows:
2016-12-21 10:13:50,816 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException
in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException


        at org.apache.hadoop.ipc.Client.call(Client.java:1411)
        at org.apache.hadoop.ipc.Client.call(Client.java:1364)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy14.blockReport(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:175)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:488)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846)
        at java.lang.Thread.run(Thread.java:744)
2016-12-21 10:13:51,256 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace:
src: /10.90.18.60:50010, dest: /10.90.18.60:39095, bytes: 24894792, op: HDFS_READ, cliID:
DFSClient_hb_rs_l-hbase40.dba.cn2.qunar.com,60020,1473253709668_1716681090_37, offset: 0,
srvID: b7cb495e-de56-4f4e-b089-abe20a3e0e53, blockid: BP-329468547-10.90.18.19-1444480193809:blk_1238808018_165078752,
duration: 1214526037


2016-12-21 10:13:51,987 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException
in offerService
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException


        at org.apache.hadoop.ipc.Client.call(Client.java:1411)
        at org.apache.hadoop.ipc.Client.call(Client.java:1364)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy14.blockReport(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.blockReport(DatanodeProtocolClientSideTranslatorPB.java:175)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.blockReport(BPServiceActor.java:488)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:711)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:846)
        at java.lang.Thread.run(Thread.java:744)




Use jstack print out the contents of regionserver as follows:
"B.defaultRpcServer.handler=62,queue=2,port=60020" daemon prio=10 tid=0x0000000000e91800 nid=0x63b8
in Object.wait() [0x00007f9b83f43000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1371)
        - locked <0x00007f9cf1334468> (a java.util.concurrent.atomic.AtomicLong)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1485)
        at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5968)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2580)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4257)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3459)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3348)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
        at qunar.tc.hbase.interceptor.support.CustomCompositeInterceptor.callMethod(CustomCompositeInterceptor.java:76)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
        at java.lang.Thread.run(Thread.java:744)



"B.defaultRpcServer.handler=61,queue=1,port=60020" daemon prio=10 tid=0x0000000000e8f800 nid=0x63b7
in Object.wait() [0x00007f9b84044000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:503)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.syncer(FSHLog.java:1371)
        - locked <0x00007f9cf1334468> (a java.util.concurrent.atomic.AtomicLong)
        at org.apache.hadoop.hbase.regionserver.wal.FSHLog.sync(FSHLog.java:1485)
        at org.apache.hadoop.hbase.regionserver.HRegion.syncOrDefer(HRegion.java:5968)
        at org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutation(HRegion.java:2580)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2261)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2213)
        at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2217)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doBatchOp(HRegionServer.java:4257)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.doNonAtomicRegionMutation(HRegionServer.java:3459)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:3348)
        at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29593)
        at qunar.tc.hbase.interceptor.support.CustomCompositeInterceptor.callMethod(CustomCompositeInterceptor.java:76)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
        at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
        at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
        at java.lang.Thread.run(Thread.java:744)



I would like to ask what is the specific reason, and how to optimize.
Mime
View raw message