hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4300) TransferFsImage.downloadEditsToStorage should use a tmp file for destination
Date Fri, 10 May 2013 21:13:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654843#comment-13654843
] 

Todd Lipcon commented on HDFS-4300:
-----------------------------------

A few small issues...

In the test:
{code}
+        new RandomAccessFile(tmpEdits[0], "rw").setLength(0);
{code}
would leak the open file, right? This might cause the test to fail on Windows where, if I
remember correctly, you can't delete a file which is open.

Might also be worth a small refactor to extract a function for this repeated code:
{code}
+        File[] tmpEdits = sd.getCurrentDir().listFiles(new FilenameFilter() {
+          @Override
+          public boolean accept(File dir, String name) {
+            return name.startsWith(NameNodeFile.EDITS_TMP.getName());
+          }
+        });
+        assertTrue(
+            "Expected a single tmp edits file in directory " + sd.toString(),
+            tmpEdits.length == 1);
{code}

Seems like there's some debug printout left:
{code}
+          System.out.println("deleteTempEdits: " + name);
{code}

Otherwise looks good.
                
> TransferFsImage.downloadEditsToStorage should use a tmp file for destination
> ----------------------------------------------------------------------------
>
>                 Key: HDFS-4300
>                 URL: https://issues.apache.org/jira/browse/HDFS-4300
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.0.2-alpha
>            Reporter: Todd Lipcon
>            Assignee: Andrew Wang
>            Priority: Critical
>         Attachments: hdfs-4300-1.patch, hdfs-4300-2.patch, hdfs-4300-3.patch, hdfs-4300-4.patch
>
>
> Currently, in TransferFsImage.downloadEditsToStorage, we download the edits file directly
to its finalized path. So, if the transfer fails in the middle, a half-written file is left
and cannot be distinguished from a correct file. So, future checkpoints by the 2NN will fail,
since the file is truncated in the middle -- but it won't ever download a good copy because
it thinks it already has the proper file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message