hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ming Ma (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-3519) Checkpoint upload may interfere with a concurrent saveNamespace
Date Fri, 09 Jan 2015 21:13:35 GMT

     [ https://issues.apache.org/jira/browse/HDFS-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ming Ma updated HDFS-3519:
    Attachment: HDFS-3519.patch

To follow up on this, https://issues.apache.org/jira/browse/HDFS-4811 discussed about adding
timestamp to checkpoint file name to manage the concurrency. Alternatively, we can move {{ImageServlet}}'s
code that handle s duplicated download requests to {{FSImage}} so that it covers regular checkpoint.
Any thoughts on this?

In addition, {{currentlyDownloadingCheckpoints}} is declared as static. That is problematic
in unit test where two namenodes share the same list. It doesn't need to be static as {{ImageServlet}}
gets the global FSImage object from {{NameNodeHttpServer}}.

Another thing about the test case, it is possible that the first standby nn checked by the
test code finishes the checkpoint and upload before the second standby nn kicks off its checkpoint
process; it doesn't appears to be the intention of the test case.  We can increase {{SlowCodec}}
sleep interval for that.

Chris, sorry didn't checkout the jira owner until after I write up the patch.

> Checkpoint upload may interfere with a concurrent saveNamespace
> ---------------------------------------------------------------
>                 Key: HDFS-3519
>                 URL: https://issues.apache.org/jira/browse/HDFS-3519
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 1.0.3, 2.0.0-alpha
>            Reporter: Todd Lipcon
>            Assignee: Chris Nauroth
>            Priority: Critical
>         Attachments: HDFS-3519.patch, test-output.txt
> TestStandbyCheckpoints failed in [precommit build 2620|https://builds.apache.org/job/PreCommit-HDFS-Build/2620//testReport/]
due to the following issue:
> - both nodes were in Standby state, and configured to checkpoint "as fast as possible"
> - NN1 starts to save its own namespace
> - NN2 starts to upload a checkpoint for the same txid. So, both threads are writing to
the same file fsimage.ckpt_12, but the actual file contents correspond to the uploading thread's
> - NN1 finished its saveNamespace operation while NN2 was still uploading. So, it renamed
the ckpt file. However, the contents of the file are still empty since NN2 hasn't sent any
> - NN2 finishes the upload, and the rename() call fails, which causes the directory to
be marked failed, etc.
> The result is that there is a file fsimage_12 which appears to be a finalized image but
in fact is incompletely transferred. When the transfer completes, the problem "heals itself"
so there wouldn't be persistent corruption unless the machine crashes at the same time. And
even then, we'd still have the earlier checkpoint to restore from.
> This same race could occur in a non-HA setup if a user puts the NN in safe mode and issues
saveNamespace operations concurrent with a 2NN checkpointing, I believe.

This message was sent by Atlassian JIRA

View raw message