hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1263) 0.20: in tryUpdateBlock, the meta file is renamed away before genstamp validation is done
Date Wed, 23 Jun 2010 08:15:49 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881608#action_12881608
] 

Todd Lipcon commented on HDFS-1263:
-----------------------------------

This is ancient code, so not sure of all the history here... but a couple things come to mind:

1) we can verify the genstamp before we do anything to the files on disk... this is probably
a good thing
2) I don't quite get why we rename to a tmp file in the first place. We shouldn't have concurrent
writers since we verified that at the top of the function, and it's synchronized so we won't
get new ones. Moving to a tmp file in the middle just creates another failure scenario where
the DN crashes in between the renames.

Can we just rename directly from the old genstamp meta to the new genstamp?

> 0.20: in tryUpdateBlock, the meta file is renamed away before genstamp validation is
done
> -----------------------------------------------------------------------------------------
>
>                 Key: HDFS-1263
>                 URL: https://issues.apache.org/jira/browse/HDFS-1263
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: data-node
>    Affects Versions: 0.20-append
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.20-append
>
>
> Saw an issue where multiple datanodes are trying to recover at the same time, and all
of them failed. I think the issue is in FSDataset.tryUpdateBlock, we do the rename of blk_B_OldGS
to blk_B_OldGS_tmpNewGS and *then* check that the generation stamp is moving upwards. Because
of this, invalid update block calls are blocked, but they then cause future updateBlock calls
to fail with "Meta file not found" errors.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message