hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1725) Set storage directories only at FSImage construction (was Cleanup FSImage construction)
Date Wed, 18 May 2011 17:30:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035506#comment-13035506

Todd Lipcon commented on HDFS-1725:

actually cancel that. I was running tests before commit and hit an issue on TestFsLimits:

        at java.util.AbstractList.add(AbstractList.java:131)
        at java.util.AbstractList.add(AbstractList.java:91)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getStorageDirs(FSNamesystem.java:425)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNamespaceDirs(FSNamesystem.java:397)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.<init>(FSImage.java:116)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.<init>(FSImage.java:104)
        at org.apache.hadoop.hdfs.server.namenode.TestFsLimits$TestFSDirectory.<init>(TestFsLimits.java:62)
        at org.apache.hadoop.hdfs.server.namenode.TestFsLimits.addChildWithName(TestFsLimits.java:153)
        at org.apache.hadoop.hdfs.server.namenode.TestFsLimits.testNoLimits(TestFsLimits.java:100)

> Set storage directories only at FSImage construction (was Cleanup FSImage construction)
> ---------------------------------------------------------------------------------------
>                 Key: HDFS-1725
>                 URL: https://issues.apache.org/jira/browse/HDFS-1725
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ivan Kelly
>            Assignee: Ivan Kelly
>             Fix For: Edit log branch (HDFS-1073)
>         Attachments: HDFS-1725-review-guide.pdf, HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff,
HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff,
> HDFS-1580 proposes extending FSEditLog to allow it to use editlog streams which are not
backed by StorageDirectory. Currently, to set the the directories used for edits, NNStorage#setStorageDirectory
is called with a list of URIs as the second argument. NNStorage takes this list or URIs, takes
all file:/// URIs and adds them to its StorageDirectory list. Then, when opened, FSEditLog
will request a list of StorageDirectories from NNStorage and create a list of EditLogOutputStreams
based on these.
> This approach cannot work with HDFS-1580. NNStorage exists solely to deal with filesystem
based storage. As such, only StorageDirectories can be retrieved from NNStorage by FSEditLog.
So, FSEditLog should get the URI from some place other than NNStorage. This presents a further
problem, in that, NNStorage#setStorageDirectories is the current way of setting the URIs for
images and edits. This call can happen at any time, so the directories in NNStorage can change
at any time. If FSEditLog is to get its URIs from elsewhere, this opens up the risk of the
filesystem directories in NNStorage and filesystem URIs being out of sync.
> A solution to this is to stipulate that the URIs for NNStorage are set only once, on
construction. All proper uses of NNStorage#setStorageDirectories are being called just after
construction of the image in any case. All other cases are using NNStorage#setStorageDirectories
not to set the storage directories, but for the side effects of this call. This guide explains
these other cases.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message