Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Mon, 4 Apr 2011 23:51:06 +0000 (UTC)
From: "Tanping Wang (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <445810151.33771.1301961066206.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (HDFS-1371) One bad node can incorrectly flag
 many files as corrupt
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015686#comment-13015686 ] 

Tanping Wang commented on HDFS-1371:
------------------------------------

I am working on this bug.  I will take Konstantin's last comment as my approach to fix this bug:

1) Add a new validate block DatanodeCommand for data node.  
2) Namenode sends the validation request to data node via heartbeat.
{code}
public DatanodeCommand[] sendHeartbeat(...)
{code}
3) data node receives the validate block command, process it and ask block scanner to validate the block.
{code}
  private boolean processCommand(DatanodeCommand cmd)
  switch(cmd.getAction()) {
      case DatanodeProtocol.DNA_VALIDATE:
       if (blockScanner != null) {
          blockScanner.verifyBlock(toVerify);
        }
  }
{code}
4) If the block is corrupted, repair the block.  If the block is good, tell name node to mark the block as good.

Any comments?

> One bad node can incorrectly flag many files as corrupt
> -------------------------------------------------------
>
>                 Key: HDFS-1371
>                 URL: https://issues.apache.org/jira/browse/HDFS-1371
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client, name-node
>    Affects Versions: 0.20.1
>         Environment: yahoo internal version 
> [knoguchi@gwgd4003 ~]$ hadoop version
> Hadoop 0.20.104.3.1007030707
>            Reporter: Koji Noguchi
>
> On our cluster, 12 files were reported as corrupt by fsck even though the replicas on the datanodes were healthy.
> Turns out that all the replicas (12 files x 3 replicas per file) were reported corrupt from one node.
> Surprisingly, these files were still readable/accessible from dfsclient (-get/-cat) without any problems.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira