hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1260) 0.20: Block lost when multiple DNs trying to recover it to different genstamps
Date Tue, 04 Oct 2011 20:48:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120459#comment-13120459

Todd Lipcon commented on HDFS-1260:

I don't believe so - I think the new append design in trunk prevents this issue. There is
an existing open JIRA against trunk about forward-porting all append-related test cases to
the new trunk implementation to be sure the new design doesn't suffer from the same issues.
> 0.20: Block lost when multiple DNs trying to recover it to different genstamps
> ------------------------------------------------------------------------------
>                 Key: HDFS-1260
>                 URL: https://issues.apache.org/jira/browse/HDFS-1260
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 0.20-append
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>             Fix For:
>         Attachments: HDFS-1260-20S.3.patch, hdfs-1260.txt, hdfs-1260.txt, simultaneous-recoveries.txt
> Saw this issue on a cluster where some ops people were doing network changes without
shutting down DNs first. So, recovery ended up getting started at multiple different DNs at
the same time, and some race condition occurred that caused a block to get permanently stuck
in recovery mode. What seems to have happened is the following:
> - FSDataset.tryUpdateBlock called with old genstamp 7091, new genstamp 7094, while the
block in the volumeMap (and on filesystem) was genstamp 7093
> - we find the block file and meta file based on block ID only, without comparing gen
> - we rename the meta file to the new genstamp _7094
> - in updateBlockMap, we do comparison in the volumeMap by oldblock *without* wildcard
GS, so it does *not* update volumeMap
> - validateBlockMetaData now fails with "blk_7739687463244048122_7094 does not exist in
blocks map"
> After this point, all future recovery attempts to that node fail in getBlockMetaDataInfo,
since it finds the _7094 gen stamp in getStoredBlock (since the meta file got renamed above)
and then fails since _7094 isn't in volumeMap in validateBlockMetadata
> Making a unit test for this is probably going to be difficult, but doable.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message