hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jitendra Nath Pandey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1725) Cleanup FSImage construction
Date Fri, 11 Mar 2011 09:36:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13005566#comment-13005566

Jitendra Nath Pandey commented on HDFS-1725:

1. setStorageDirectories also calls removedStorageDirs.clear() and re-initializes storage
directories. This patch removes the calls to setStorageDirectories from a few places, therefore
those StorageDirectories that were removed due to some error might continue to hang in removedStorageDirs
and won't be reinstated. Will attemptRestoreRemovedStorage take care of clearing up of removedStorageDirs
in all cases?

 > private FSImage(Configuration conf, FSNamesystem ns) throws IOException {
 >    this(conf, ns, (Collection<URI>)null, (Collection<URI>)null);
 >  }

Can we change the call to the constructor as
    this(conf, ns, FSNamesystem.getNamespaceDirs(conf),
          FSNamesystem.getNamespaceEditsDirs(conf))  instead of passing null,
   and remove the check  imageDirs==null in the constructor body?

>+    for (URI uri : editDirsToFormat) {
>+      if (!dirsToFormat.contains(uri)) {
>+        dirsToFormat.add(uri);
>+      }
>+    }
  This means currently we don't ask for confirmation before formatting edit directories that
are not namespace directories. I am wondering do we need to change that, although it seems
to be ok.

4. In SecondaryNameNode.java#startCheckpoint.
   What is the reason behind removing the call to unlockAll?

   In recoverCreate a call to unlockAll is added and storage.close is removed. storage.close
was also calling listeners.clear, which will not be called now. Is that ok?

5. Javadoc needs to be fixed for some of the methods with changed signatures, for example
recoverCreateRead in BackupImage.java.

> Cleanup FSImage construction
> ----------------------------
>                 Key: HDFS-1725
>                 URL: https://issues.apache.org/jira/browse/HDFS-1725
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ivan Kelly
>            Assignee: Ivan Kelly
>             Fix For: 0.23.0
>         Attachments: HDFS-1725.diff, HDFS-1725.diff
> FSImage construction is messy. Sometimes the storagedirectories in use are set straight
away, sometimes they are not. This makes it hard for anything under FSImage (i.e. FSEditLog)
to make assumptions about what it can use. Therefore, this patch makes FSImage set the storage
directories in use during construction, and never allows them to change. If you want to change
storagedirectories you create a new image.
> Also, all the construction code should be the same with the only difference being the
parameters passed. When not passed, these should get sensible defaults.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message