hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengbing Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8113) NullPointerException in BlockInfoContiguous causes block report failure
Date Fri, 17 Apr 2015 10:25:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499591#comment-14499591
] 

Chengbing Liu commented on HDFS-8113:
-------------------------------------

Thanks [~vinayrpet] for your advice! I got the following debug logs.
{quote}
2015-04-17 15:38:54,801 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
Reported block blk_1143745403_70011665 on 10.153.80.84:1004 size 2631763 replicaState = FINALIZED
2015-04-17 15:38:54,801 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
In memory blockUCState = COMPLETE
2015-04-17 15:38:54,801 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
Reported block blk_1185006557_111278782 on 10.153.80.84:1004 size 19005434 replicaState =
FINALIZED
2015-04-17 15:38:54,801 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
Reported block blk_1189413471_115690616 on 10.153.80.84:1004 size 99678737 replicaState =
FINALIZED
2015-04-17 15:38:54,801 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
Reported block blk_1171261663_97530254 on 10.153.80.84:1004 size 13847 replicaState = FINALIZED
2015-04-17 15:38:54,801 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
Reported block blk_1149751102_76017688 on 10.153.80.84:1004 size 6702 replicaState = FINALIZED
2015-04-17 15:38:54,801 DEBUG org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
In memory blockUCState = COMPLETE
2015-04-17 15:38:54,801 WARN org.apache.hadoop.ipc.Server: IPC Server handler 109 on 8020,
call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.blockReport from 10.153.80.84:38504
Call#4258262 R
etry#0
java.lang.NullPointerException
{quote}

The stacktrace is missing due to JVM default optimization... OmitStackTraceInFastThrow is
the default option, and I didn't unset it. It will recompile a method if it has thrown some
exception too many times. The stacktrace in the issue description is got from a DN a month
ago. 

>From the above logs, it is a FINALIZED block in a report that caused the NPE. So the stacktrace
in the description is incorrect. Really sorry for that.

Then I checked the last block blk_1149751102_76017688 with oiv against fsimage. The file is
OK. I can download it through FS shell. I also checked all three DNs containing this block,
and they all have the same file, genstamp and meta. It seems the active NameNode's holding
incorrect information on this block.

> NullPointerException in BlockInfoContiguous causes block report failure
> -----------------------------------------------------------------------
>
>                 Key: HDFS-8113
>                 URL: https://issues.apache.org/jira/browse/HDFS-8113
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.6.0
>            Reporter: Chengbing Liu
>            Assignee: Chengbing Liu
>         Attachments: HDFS-8113.02.patch, HDFS-8113.patch
>
>
> The following copy constructor can throw NullPointerException if {{bc}} is null.
> {code}
>   protected BlockInfoContiguous(BlockInfoContiguous from) {
>     this(from, from.bc.getBlockReplication());
>     this.bc = from.bc;
>   }
> {code}
> We have observed that some DataNodes keeps failing doing block reports with NameNode.
The stacktrace is as follows. Though we are not using the latest version, the problem still
exists.
> {quote}
> 2015-03-08 19:28:13,442 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: RemoteException
in offerService
> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockInfo.(BlockInfo.java:80)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager$BlockToMarkCorrupt.(BlockManager.java:1696)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.checkReplicaCorrupt(BlockManager.java:2185)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReportedBlock(BlockManager.java:2047)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.reportDiff(BlockManager.java:1950)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1823)
> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.processReport(BlockManager.java:1750)
> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.blockReport(NameNodeRpcServer.java:1069)
> at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.blockReport(DatanodeProtocolServerSideTranslatorPB.java:152)
> at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:26382)
> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1623)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message