hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1994) Fix race conditions when running two rapidly checkpointing 2NNs
Date Wed, 25 May 2011 01:24:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038909#comment-13038909
] 

Todd Lipcon commented on HDFS-1994:
-----------------------------------

issue #1 is fairly straightforward to address - just need to write to a side file and rename
into place

the issue #2 is a bit subtle. The interleaving is the following:

2NN A) calls rollEdits()
2NN B) calls rollEdits()
2NN A) calls getRemoteEditLogManifest()
2NN B) calls getRemoteEditLogManifest()

both of them then see the same manifest, and download all of the edits available. Therefore
they merge to the same transaction.

This can be fixed one of two ways:
a) add a lock to the code that downloads an image, to make sure that only one image with a
certain txid could be downloaded at a time
b) when the 2NN calls rollEdits, it should only try to checkpoint up to the transaction where
the roll happened.

IMO we should actually do both: (a) as a sanity check, and (b) to ensure each checkpoint is
unique, thus allowing us freedom later to do something like include the 2NN's hostname in
the image itself

> Fix race conditions when running two rapidly checkpointing 2NNs
> ---------------------------------------------------------------
>
>                 Key: HDFS-1994
>                 URL: https://issues.apache.org/jira/browse/HDFS-1994
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: name-node
>    Affects Versions: Edit log branch (HDFS-1073)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: Edit log branch (HDFS-1073)
>
>         Attachments: hdfs-1994.txt
>
>
> HDFS-1984 added the ability to run two secondary namenodes at the same time. However,
there were two races I found when stress testing this (by running two NNs each checkpointing
in a tight loop with no sleep):
> 1) the writing of the seen_txid file was not atomic, so it was at some points reading
an empty file
> 2) it was possible for two checkpointers to try to take a checkpoint at the same transaction
ID, which would cause the two image downloads to collide and fail

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message