hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sam rash (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1263) 0.20: in tryUpdateBlock, the meta file is renamed away before genstamp validation is done
Date Wed, 23 Jun 2010 16:45:51 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881739#action_12881739
] 

sam rash commented on HDFS-1263:
--------------------------------

several thoughts/comments:

my reading of the code is that the temp file was to make the creating of a meta file that
is both truncated and has the new genstamp and atomic operation on the filesystem.  If we
rename first and crash and recover, then how do we know that the truncation didn't finish?
(without information from the NN or other node giving us a new length).  If we truncate first,
then we have effectively corrupted a block.

can you also explain the error state that results?  (truncated blocks, infinite loops, bad
meta-data, etc)

and do i follow that a client started 2 lease recoveries?  or was this a client and a NN somehow?
 (ie, how were there concurrent recoveries of the same block)

seems like extra synchronization in parts of updateBlock might help as well.

also, we check the genstamp is moving upwards both at the start of updateBlock and at the
end of tryUpdateBlock.  do you know why?

> 0.20: in tryUpdateBlock, the meta file is renamed away before genstamp validation is
done
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-1263
>                 URL: https://issues.apache.org/jira/browse/HDFS-1263
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20-append
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.20-append
>
>
> Saw an issue where multiple datanodes are trying to recover at the same time, and all
of them failed. I think the issue is in FSDataset.tryUpdateBlock, we do the rename of blk_B_OldGS
to blk_B_OldGS_tmpNewGS and *then* check that the generation stamp is moving upwards. Because
of this, invalid update block calls are blocked, but they then cause future updateBlock calls
to fail with "Meta file not found" errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message