hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3122) Block recovery with closeFile flag true can race with blockReport. Due to this blocks are getting marked as corrupt.
Date Thu, 28 Jun 2012 16:29:44 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403200#comment-13403200
] 

Uma Maheswara Rao G commented on HDFS-3122:
-------------------------------------------

Thanks a lot Nicholas.

{quote}
- DN side: DN receives a replica-delete with the older genstamp from NN but the stored genstamp
is newer.  So it ignores the replica-delete.
{quote}
In my case, this happened for all the blocks unfortunately. I have seen the logs, all blocks
marked as corrupt almost at same time. So, it will just leave all the corrupt block as is.

 Yes, here is one bug I remember. HDFS-3157. NN will add the block as corrupt with NN genstamp
rather than reported genstamp ( I will take a look on that patch again). This should solve
this problem. But all DNs configured BR time same and happens for all means, all blocks will
get marked as corrupt right?

                
> Block recovery with closeFile flag true can race with blockReport. Due to this blocks
are getting marked as corrupt.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-3122
>                 URL: https://issues.apache.org/jira/browse/HDFS-3122
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node, name-node
>    Affects Versions: 0.23.0, 0.24.0
>            Reporter: Uma Maheswara Rao G
>            Assignee: Uma Maheswara Rao G
>            Priority: Critical
>         Attachments: blockCorrupt.txt
>
>
> *Block Report* can *race* with *Block Recovery* with closeFile flag true.
>  Block report generated just before block recovery at DN side and due to N/W problems,
block report got delayed to NN. 
> After this, recovery success and generation stamp modifies to new one. 
> And primary DN invokes the commitBlockSynchronization and block got updated in NN side.
Also block got marked as complete, since the closeFile flag was true. Updated with new genstamp.
> Now blockReport started processing at NN side. This particular block from RBW (when it
generated the BR at DN), and file was completed at NN side.
> Finally block will be marked as corrupt because of genstamp mismatch.
> {code}
>  case RWR:
>       if (!storedBlock.isComplete()) {
>         return null; // not corrupt
>       } else if (storedBlock.getGenerationStamp() != iblk.getGenerationStamp()) {
>         return new BlockToMarkCorrupt(storedBlock,
>             "reported " + reportedState + " replica with genstamp " +
>             iblk.getGenerationStamp() + " does not match COMPLETE block's " +
>             "genstamp in block map " + storedBlock.getGenerationStamp());
>       } else { // COMPLETE block, same genstamp
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message