hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1218) 20 append: Blocks recovered on startup should be treated with lower priority during block synchronization
Date Wed, 28 Mar 2012 11:42:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240355#comment-13240355
] 

Uma Maheswara Rao G commented on HDFS-1218:
-------------------------------------------

{code}
 if (!shouldRecoverRwrs && info.wasRecoveredOnStartup()) {
+          LOG.info("Not recovering replica " + record + " since it was recovered on "
+              + "startup and we have better replicas");
+          continue;
+        }{code}

@Todd, can you please tell why this check added?

Take a case,  

 1) DN1->DN2->DN3 are in pipeline.
 2) Client killed abruptly
 3) one DN has restarted , say DN3
 4) In DN3 info.wasRecoveredOnStartup() will be true
 5) NN recovery triggered, DN3 skipped from recovery due to above check.
 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say
1 and also DN3 still has this block entry in ongoingCreates
 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2)
 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation
stamp.
 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to
blocksBeingWritten directory.
 
When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId
entry still presents in ongoingCreates with startup recovery true.
It may keep continue this dance for evry recovery.
And this stale replica will not be cleaned untill we restart the cluster. Actual replica will
be trasferred to this node only through replication process.

Also unnecessarily that replicated blocks will get invalidated after next recoveries....

I understood that check might be because to exclude the restarted node for calculating the
min lengths to truncate.

                
> 20 append: Blocks recovered on startup should be treated with lower priority during block
synchronization
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-1218
>                 URL: https://issues.apache.org/jira/browse/HDFS-1218
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20-append
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For: 0.20.205.0
>
>         Attachments: HDFS-1218.20s.2.patch, hdfs-1281.txt
>
>
> When a datanode experiences power loss, it can come back up with truncated replicas (due
to local FS journal replay). Those replicas should not be allowed to truncate the block during
block synchronization if there are other replicas from DNs that have _not_ restarted.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message