commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Roger Whitcomb (JIRA)" <>
Subject [jira] [Commented] (VFS-555) Add ability to specify alternate HDFS configuration file
Date Thu, 08 Jan 2015 17:59:35 GMT


Roger Whitcomb commented on VFS-555:

Thanks, Bernd!  My reasoning on the log level is that loading an alternate configuration is
a fairly significant event in the HDFS lifecycle, so it should be given some prominence. 
But I'm happy to reduce the level to debug also.
My understanding of the way this works is that any settings in this alternate config file
will override the defaults (which are specified in some "hdfs-xxx.xml" file in one of the
hdfs .jar files.  This FS_DEFAULT_NAME_KEY is independent of that (being the default location
we're connecting to, I believe).  In our case, the default value is "dfs.client.use.datanode.hostname=false",
so ours overrides the default.  But, I believe the VFS configuration that we're dealing with
(and that FS_DEFAULT_NAME_KEY refers to) is different than this HDFS configuration.  I know
it's confusing because they are both called "configuration".  So, I don't think the order
of these statements matters, but in case it does, I think it is safer to leave the new code
AFTER the existing code.  Does that make sense?

> Add ability to specify alternate HDFS configuration file
> --------------------------------------------------------
>                 Key: VFS-555
>                 URL:
>             Project: Commons VFS
>          Issue Type: Improvement
>    Affects Versions: 2.1
>         Environment: All
>            Reporter: Roger Whitcomb
>            Priority: Minor
>             Fix For: 2.1
>         Attachments: VFS-555.patch
> We need to access HDFS on an AWS installation from outside the AWS firewall.  The technique
is described here:
> In order to do this we need to set the "dfs.client.use.datanode.hostname=true" in an
alternate configuration file set after the default configuration is loaded.
> I propose to add a property "configName" to HdfsFileSystemConfigBuilder, and then code
in HdfsFileSystem to get the value of the property and set it in the Configuration.

This message was sent by Atlassian JIRA

View raw message