hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4300) TransferFsImage.downloadEditsToStorage should use a tmp file for destination
Date Fri, 10 May 2013 21:13:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654843#comment-13654843

Todd Lipcon commented on HDFS-4300:

A few small issues...

In the test:
+        new RandomAccessFile(tmpEdits[0], "rw").setLength(0);
would leak the open file, right? This might cause the test to fail on Windows where, if I
remember correctly, you can't delete a file which is open.

Might also be worth a small refactor to extract a function for this repeated code:
+        File[] tmpEdits = sd.getCurrentDir().listFiles(new FilenameFilter() {
+          @Override
+          public boolean accept(File dir, String name) {
+            return name.startsWith(NameNodeFile.EDITS_TMP.getName());
+          }
+        });
+        assertTrue(
+            "Expected a single tmp edits file in directory " + sd.toString(),
+            tmpEdits.length == 1);

Seems like there's some debug printout left:
+          System.out.println("deleteTempEdits: " + name);

Otherwise looks good.
> TransferFsImage.downloadEditsToStorage should use a tmp file for destination
> ----------------------------------------------------------------------------
>                 Key: HDFS-4300
>                 URL: https://issues.apache.org/jira/browse/HDFS-4300
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.0.2-alpha
>            Reporter: Todd Lipcon
>            Assignee: Andrew Wang
>            Priority: Critical
>         Attachments: hdfs-4300-1.patch, hdfs-4300-2.patch, hdfs-4300-3.patch, hdfs-4300-4.patch
> Currently, in TransferFsImage.downloadEditsToStorage, we download the edits file directly
to its finalized path. So, if the transfer fails in the middle, a half-written file is left
and cannot be distinguished from a correct file. So, future checkpoints by the 2NN will fail,
since the file is truncated in the middle -- but it won't ever download a good copy because
it thinks it already has the proper file.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message