hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5605) All the replicas incorrectly got marked as corrupt.
Date Wed, 01 Apr 2009 00:35:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12694334#action_12694334
] 

Raghu Angadi commented on HADOOP-5605:
--------------------------------------

I think it is simpler to just describe what happened :

 * Some datanode went down and X and Y are the remaining replicas for B (of length 53384156)
.
 * NN asked X to replicate the block B  to P
 * P reported B as corrupt since CRC failed on its side.
 * NN marks B on X as corrupt.
 * NN asks Y to replicate  B to Q
 * Y reports the following error in the log and reports the block as corrupt.
    ** _Can't replicate block B because on-disk length 53384156 is shorter than NameNode recorded
length 134217728_
 * This mismatch happened since when P reported the first corruption, it used a block object
with default length of 128 MB, and NN incorrectly keeps that object in its neededReplication
queue.

This affects only 0.20 since 0.20 started reporting length and CRC mismatches during replication
attempts.



> All the replicas incorrectly got marked as corrupt.
> ---------------------------------------------------
>
>                 Key: HADOOP-5605
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5605
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.20.0
>            Reporter: Raghu Angadi
>             Fix For: 0.2.0
>
>
> NameNode does not handle {{reportBadBlocks()}} properly. As a result, when DataNode reports
the corruption (only in the case of block transfer between two datanodes), further attempts
to replicate the block end up marking all the replicas as corrupt!
> From the implementation, it looks like NN incorrectly uses the block object used in RPC
to queue to neededReplication queue instead of using internal block object. 
> will include an actual example in the next comment.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message