hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-3519) Checkpoint upload may interfere with a concurrent saveNamespace
Date Fri, 08 Jun 2012 22:48:23 GMT
Todd Lipcon created HDFS-3519:

             Summary: Checkpoint upload may interfere with a concurrent saveNamespace
                 Key: HDFS-3519
                 URL: https://issues.apache.org/jira/browse/HDFS-3519
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: name-node
    Affects Versions: 2.0.0-alpha, 1.0.3
            Reporter: Todd Lipcon
            Assignee: Todd Lipcon
            Priority: Critical

TestStandbyCheckpoints failed in [precommit build 2620|https://builds.apache.org/job/PreCommit-HDFS-Build/2620//testReport/]
due to the following issue:
- both nodes were in Standby state, and configured to checkpoint "as fast as possible"
- NN1 starts to save its own namespace
- NN2 starts to upload a checkpoint for the same txid. So, both threads are writing to the
same file fsimage.ckpt_12, but the actual file contents correspond to the uploading thread's
- NN1 finished its saveNamespace operation while NN2 was still uploading. So, it renamed the
ckpt file. However, the contents of the file are still empty since NN2 hasn't sent any bytes
- NN2 finishes the upload, and the rename() call fails, which causes the directory to be marked
failed, etc.

The result is that there is a file fsimage_12 which appears to be a finalized image but in
fact is incompletely transferred. When the transfer completes, the problem "heals itself"
so there wouldn't be persistent corruption unless the machine crashes at the same time. And
even then, we'd still have the earlier checkpoint to restore from.

This same race could occur in a non-HA setup if a user puts the NN in safe mode and issues
saveNamespace operations concurrent with a 2NN checkpointing, I believe.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message