hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1725) Cleanup FSImage construction
Date Sun, 03 Apr 2011 21:21:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13015224#comment-13015224
] 

Ivan Kelly commented on HDFS-1725:
----------------------------------

Just to clarify the issue I'm trying to address with this JIRA while it's fresh in my mind
(just hit the same problem again on HDFS-1580). In HDFS-1580 I'm trying to add the ability
to have a editlog output stream which is not backed by a StorageDirectory. Currently images
and edits are both stored in a StorageDirectory. 

Currently when FSImage and FSEditLog are created, the image and edit URIs may or may not be
set. If they are not set on construction, NNStorage.setStorageDirectories is called to set
them. This isn't a problem because FSEditLog only uses StorageDirectory backed output streams
and as such can get the list of locations to write to from storage, which is exactly what
it does. On FSEditLog.open() is iterates through all EDITS type directories in the storage
object, and creates a stream for it.

Obviously this will not work for anything but StorageDirectories and it's precisely this problem
which I was trying to address here. If NNStorage.setStorageDirectories() is used, you have
to go to NNStorage for the location list when calling FSEditLog.open as the list could change
at any time. The solution I proposed was to mandate that you always set the StorageDirectories
on construction. This means that FSEditLog can get the list on construction from FSImage and
never has to query NNStorage. The only other solution I can think of is to add a setStorageDirectories
call to FSImage which calls the same call on NNStorage and sets the list for FSEditLog. This
would work, but does seem as elegant as just setting them on construction.

> Cleanup FSImage construction
> ----------------------------
>
>                 Key: HDFS-1725
>                 URL: https://issues.apache.org/jira/browse/HDFS-1725
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ivan Kelly
>            Assignee: Ivan Kelly
>             Fix For: 0.23.0
>
>         Attachments: HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff, HDFS-1725.diff,
HDFS-1725.diff, HDFS-1725.diff
>
>
> FSImage construction is messy. Sometimes the storagedirectories in use are set straight
away, sometimes they are not. This makes it hard for anything under FSImage (i.e. FSEditLog)
to make assumptions about what it can use. Therefore, this patch makes FSImage set the storage
directories in use during construction, and never allows them to change. If you want to change
storagedirectories you create a new image.
> Also, all the construction code should be the same with the only difference being the
parameters passed. When not passed, these should get sensible defaults.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message