hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2305) Running multiple 2NNs can result in corrupt file system
Date Wed, 31 Aug 2011 21:28:10 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094912#comment-13094912

Aaron T. Myers commented on HDFS-2305:

There are two obvious work-arounds for this issue:

# Explicitly configure the address of the 2NN ({dfs.secondary.http.address}). This would prevent
2NNs from starting up which couldn't bind to that address.
# Do something else to make sure that there is only ever one 2NN running.

But, we should still harden HDFS to make it so that this scenario is less likely to occur.
Right now it's all too easy (with the default configs) to find oneself in this scenario.

I can think of a few possible solutions:

# Don't have a default value for the {{dfs.secondary.http.address}}. Require the user set
it, and don't allow the 2NN to start up without it. The NN will reject connections to roll/fetch
fsimage/edits from any machine that's not connecting from this configured address.
# On start-up, the 2NN makes an RPC to the NN to generate a unique token. This token is subsequently
used for all NN and 2NN communication. The NN will reject any communication from a 2NN with
a different token. This will effectively lock out any previously-started 2NNs from mutating
the NN state.
# Before transferring the fsimage back to the NN, the 2NN computes a checksum of the newly-merged
fsimage, and informs the NN of the expected checksum. On download of the new fsimage, the
NN verifies the checksum of the downloaded file against the expected checksum from the 2NN.

Of these, I think I'm inclined to go with option 3. Option 1 is dead simple, but has the downside
of changing default config options and requiring an extra step to set up a Hadoop cluster.
Option 2 seems like overkill to me. Option 3 is relatively simple, and has the added benefit
of providing an extra integrity check of the fsimage state during network transfer.


> Running multiple 2NNs can result in corrupt file system
> -------------------------------------------------------
>                 Key: HDFS-2305
>                 URL: https://issues.apache.org/jira/browse/HDFS-2305
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.20.2
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
> Here's the scenario:
> * You run the NN and 2NN (2NN A) on the same machine.
> * You don't have the address of the 2NN configured, so it's defaulting to
> * There's another 2NN (2NN B) running on a second machine.
> * When a 2NN is done checkpointing, it says "hey NN, I have an updated fsimage for you.
You can download it from this URL, which includes my IP address, which is x"
> And here's the steps that occur to cause this issue:
> # Some edits happen.
> # 2NN A (on the NN machine) does a checkpoint. All is dandy.
> # Some more edits happen.
> # 2NN B (on a different machine) does a checkpoint. It tells the NN "grab the newly-merged
fsimage file from"
> # NN happily grabs the fsimage from 2NN A (the 2NN on the NN machine), which is stale.
> # NN renames edits.new file to edits. At this point the in-memory FS state is fine, but
the on-disk state is missing edits.
> # The next time a 2NN (any 2NN) tries to do a checkpoint, it gets an up-to-date edits
file, with an outdated fsimage, and tries to apply those edits to that fsimage.
> # Kaboom.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message