hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4820) Remove hdfs-default.xml
Date Fri, 17 May 2013 16:55:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13660852#comment-13660852
] 

Chris Nauroth commented on HDFS-4820:
-------------------------------------

Removing *-default.xml seems to complicate resolution of some of our more dynamic configuration
properties.  A good example is our mapping of file system impl classes by URI scheme used
by {{AbstractFileSystem#createFileSystem}}:

{code}
<property>
  <name>fs.AbstractFileSystem.file.impl</name>
  <value>org.apache.hadoop.fs.local.LocalFs</value>
  <description>The AbstractFileSystem for file: uris.</description>
</property>


<property>
  <name>fs.AbstractFileSystem.hdfs.impl</name>
  <value>org.apache.hadoop.fs.Hdfs</value>
  <description>The FileSystem for hdfs: uris.</description>
</property>
{code}

{code}
  public static AbstractFileSystem createFileSystem(URI uri, Configuration conf)
      throws UnsupportedFileSystemException {
    Class<?> clazz = conf.getClass("fs.AbstractFileSystem." + 
                                uri.getScheme() + ".impl", null);
    if (clazz == null) {
      throw new UnsupportedFileSystemException(
          "No AbstractFileSystem for scheme: " + uri.getScheme());
    }
    return (AbstractFileSystem) newInstance(clazz, uri, conf);
  }
{code}

Without defaults in the XML, this code will need to hard-code the mapping somewhere.  We'll
have to remember to cover all cases like this.

{quote}
...it should not be part of the jar and should not be looked for and loaded in by default
into the Configuration object.
{quote}

This may be a bigger concern for compatibility.  {{Configuration}} is annotated public/stable,
and I've seen a lot of tutorials with sample code that instantiates a new instance and expects
it to be fully populated with the keys from *-default.xml.  For full compatibility, I suppose
we'd need to update not only our own {{Configuration#get}} calls to enforce the defaults,
but also guarantee that if a client creates a new instance, they get the same values that
used to be provided in the XML.  Again, this probably would involve some kind of hard-coding
during static initialization.
                
> Remove hdfs-default.xml
> -----------------------
>
>                 Key: HDFS-4820
>                 URL: https://issues.apache.org/jira/browse/HDFS-4820
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.0.4-alpha
>            Reporter: Siddharth Seth
>
> Similar to YARN-673, which contains additional details.
> There's separate jiras for YARN, MR and HDFS so enough people take a look. Looking for
reasons for these files to exist, other than the ones mentioned in YARN-673, or a good reason
to keep the files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message