hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Bockelman (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4402) Namenode is unaware of FS corruption
Date Mon, 20 Oct 2008 21:39:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641166#action_12641166
] 

Brian Bockelman commented on HADOOP-4402:
-----------------------------------------

Ah - mystery solved.  It turns out that there actually was a large-scale corruption event
that took out half the hadoop data nodes, but no one realized it until the above recurred.

We have a "scratch cleanup" script which did a "rm" on all of the hadoop data directories
for about 100 nodes.  So, starting at midnight, all the block verification attempts started
failing because there were no blocks to read; when I restarted the namenode, the data nodes
suddenly had new block reports saying they all had no blocks.

Please close.

> Namenode is unaware of FS corruption
> ------------------------------------
>
>                 Key: HADOOP-4402
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4402
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.18.1
>            Reporter: Brian Bockelman
>             Fix For: 0.18.2
>
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> I think the name node is not told when there is block corruption.
> I found a huge number of files corrupted when I restarted my namenode.  Digging through
the datanode logs, I saw the following:
> 2008-10-13 03:30:44,266 INFO org.apache.hadoop.dfs.DataBlockScanner: Reporting bad block
blk_-54103619973430645_3038 to namenode.
> 2008-10-13 03:57:11,447 WARN org.apache.hadoop.dfs.DataBlockScanner: First Verification
failed for blk_7657563767222456337_3165. Exception : java.io.IOException: Block blk_7657563767222456337_3165
is not valid.
> 	at org.apache.hadoop.dfs.FSDataset.getBlockFile(FSDataset.java:716)
> 	at org.apache.hadoop.dfs.FSDataset.getLength(FSDataset.java:704)
> 	at org.apache.hadoop.dfs.DataNode$BlockSender.<init>(DataNode.java:1678)
> 	at org.apache.hadoop.dfs.DataBlockScanner.verifyBlock(DataBlockScanner.java:408)
> 	at org.apache.hadoop.dfs.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:474)
> 	at org.apache.hadoop.dfs.DataBlockScanner.run(DataBlockScanner.java:565)
> 	at java.lang.Thread.run(Thread.java:595)
> 2008-10-13 03:57:11,448 WARN org.apache.hadoop.dfs.DataBlockScanner: Second Verification
failed for blk_7657563767222456337_3165. Exception : java.io.IOException: Block blk_7657563767222456337_3165
is not valid.
> 	at org.apache.hadoop.dfs.FSDataset.getBlockFile(FSDataset.java:716)
> 	at org.apache.hadoop.dfs.FSDataset.getLength(FSDataset.java:704)
> 	at org.apache.hadoop.dfs.DataNode$BlockSender.<init>(DataNode.java:1678)
> 	at org.apache.hadoop.dfs.DataBlockScanner.verifyBlock(DataBlockScanner.java:408)
> 	at org.apache.hadoop.dfs.DataBlockScanner.verifyFirstBlock(DataBlockScanner.java:474)
> 	at org.apache.hadoop.dfs.DataBlockScanner.run(DataBlockScanner.java:565)
> 	at java.lang.Thread.run(Thread.java:595)
> 2008-10-13 03:57:11,448 INFO org.apache.hadoop.dfs.DataBlockScanner: Reporting bad block
blk_7657563767222456337_3165 to namenode.
> So, node099 found a bad block.  However, if I grep the namenode information for that
block:
> /scratch/hadoop/logs/hadoop-root-namenode-hadoop-name.log.2008-10-10:2008-10-10 20:21:20,002
INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.allocateBlock: /user/uscms01/LoadTestDownload/LoadTest07_FNAL_01_MyQXiu5a22TJQlcB_508.
blk_7657563767222456337_3165
> /scratch/hadoop/logs/hadoop-root-namenode-hadoop-name.log.2008-10-10:2008-10-10 20:21:32,150
INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated:
172.16.1.110:50010 is added to blk_7657563767222456337_3165 size 67108864
> /scratch/hadoop/logs/hadoop-root-namenode-hadoop-name.log.2008-10-10:2008-10-10 20:21:32,151
INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated:
172.16.1.99:50010 is added to blk_7657563767222456337_3165 size 67108864
> /scratch/hadoop/logs/hadoop-root-namenode-hadoop-name.log.2008-10-12:2008-10-12 05:05:26,898
INFO org.apache.hadoop.dfs.StateChange: BLOCK* ask 172.16.1.99:50010 to replicate blk_7657563767222456337_3165
to datanode(s) 172.16.1.18:50010
> /scratch/hadoop/logs/hadoop-root-namenode-hadoop-name.log.2008-10-12:2008-10-12 05:05:40,742
INFO org.apache.hadoop.dfs.NameNode: Error report from 172.16.1.99:50010: Can't send invalid
block blk_7657563767222456337_3165
> /scratch/hadoop/logs/hadoop-root-namenode-hadoop-name.log.2008-10-12:2008-10-12 05:12:43,759
WARN org.apache.hadoop.fs.FSNamesystem: PendingReplicationMonitor timed out block blk_7657563767222456337_3165
> To summarize:
> - Block is allocated and written successfully to node100, then replicated to node099.
> - Name node asks node099 to replicate block to node018
> - Name node is told it can't send invalid block to node018!  A few minutes later, the
PendingReplicationMonitor times out
> - No new replications are launched!!!
> - Block is found to be corrupted on node099 a few days later.  Data node claims to inform
the namenode of this, but nothing is listed in the namenode logs.
> - Block is suspiciously missing on node110 as well
> Perhaps there are a few bugs here?
> 1) Name node doesn't get notified of the corrupted blocks - even though the datanode
claims to!
> 2) On replication failure, no new replicas are created.
> 3) Corruption events are much, much too common.  We have a specific dataset which the
namenode now claims is mostly-corrupted (100/167 files); before I restarted the namenode,
we had jobs run against it continuously for the entire weekend.  The jobs were all successful,
and the binary data format does internal integrity checks as it reads the files.  If the corruption
was real, jobs would have failed.
> I'm concerned that the corruption-detection systems of Hadoop are seriously busted for
me. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message