hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wendy Chien (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-855) HDFS should repair corrupted files
Date Wed, 10 Jan 2007 23:58:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12463770

Wendy Chien commented on HADOOP-855:

Thanks Doug and Hairong for looking over the patch. 

a. I'll change the name from deleteBlocks to invalidateBlocks
b. I'll take away the one location enforcement in NameNode
c. I'm going to call removeStoredBlock to update the data structures in invalidateBlock. 
We need to do everything it does.  (It updates blocksMap, neededReplications, and excessReplicateMap)
d. I agree with Hairong that we should not delete the corrupt copy if it is the only one on
a live node, in case the decommissioned nodes are taken down.  I will implement it this way
unless people disagree. 

> HDFS should repair corrupted files
> ----------------------------------
>                 Key: HADOOP-855
>                 URL: https://issues.apache.org/jira/browse/HADOOP-855
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>            Reporter: Wendy Chien
>         Assigned To: Wendy Chien
>         Attachments: hadoop-855-5.patch
> While reading if we discover a mismatch between a block and checksum, we want to report
this back to the namenode to delete the corrupted block or crc.
> To implement this, we need to do the following:
> DFSInputStream
> 1. move DFSInputStream out of DFSClient
> 2. add member variable to keep track of current datanode (the chosen node)
> DistributedFileSystem
> 1. change reportChecksumFailure parameter crc from int to FSInputStream (needed to be
able to delete it). 
> 2. determine specific block and datanode from DFSInputStream passed to reportChecksumFailure
> 3. call namenode to delete block/crc vis DFSClient
> ClientProtocol
> 1. add method to ask namenode to delete certain blocks on specifc datanode.
> Namenode
> 1. add ability to delete certain blocks on specific datanode

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message