hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tanping Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1371) One bad node can incorrectly flag many files as corrupt
Date Mon, 04 Apr 2011 23:51:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015686#comment-13015686
] 

Tanping Wang commented on HDFS-1371:
------------------------------------

I am working on this bug.  I will take Konstantin's last comment as my approach to fix this
bug:

1) Add a new validate block DatanodeCommand for data node.  
2) Namenode sends the validation request to data node via heartbeat.
{code}
public DatanodeCommand[] sendHeartbeat(...)
{code}
3) data node receives the validate block command, process it and ask block scanner to validate
the block.
{code}
  private boolean processCommand(DatanodeCommand cmd)
  switch(cmd.getAction()) {
      case DatanodeProtocol.DNA_VALIDATE:
       if (blockScanner != null) {
          blockScanner.verifyBlock(toVerify);
        }
  }
{code}
4) If the block is corrupted, repair the block.  If the block is good, tell name node to mark
the block as good.

Any comments?

> One bad node can incorrectly flag many files as corrupt
> -------------------------------------------------------
>
>                 Key: HDFS-1371
>                 URL: https://issues.apache.org/jira/browse/HDFS-1371
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client, name-node
>    Affects Versions: 0.20.1
>         Environment: yahoo internal version 
> [knoguchi@gwgd4003 ~]$ hadoop version
> Hadoop 0.20.104.3.1007030707
>            Reporter: Koji Noguchi
>
> On our cluster, 12 files were reported as corrupt by fsck even though the replicas on
the datanodes were healthy.
> Turns out that all the replicas (12 files x 3 replicas per file) were reported corrupt
from one node.
> Surprisingly, these files were still readable/accessible from dfsclient (-get/-cat) without
any problems.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message