metamodel-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kasper Sørensen (JIRA) <j...@apache.org>
Subject [jira] [Resolved] (METAMODEL-219) HdfsResource's FileSystem should be createable based on custom core-site.xml and hdfs-site.xml files
Date Sat, 12 Dec 2015 21:49:46 GMT

     [ https://issues.apache.org/jira/browse/METAMODEL-219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kasper Sørensen resolved METAMODEL-219.
---------------------------------------
    Resolution: Fixed
      Assignee: Kasper Sørensen

Fixed via https://github.com/apache/metamodel/pull/78

> HdfsResource's FileSystem should be createable based on custom core-site.xml and hdfs-site.xml
files
> ----------------------------------------------------------------------------------------------------
>
>                 Key: METAMODEL-219
>                 URL: https://issues.apache.org/jira/browse/METAMODEL-219
>             Project: Apache MetaModel
>          Issue Type: Improvement
>    Affects Versions: 4.4.1
>            Reporter: Kasper Sørensen
>            Assignee: Kasper Sørensen
>
> Today we always assume a very vanilla HDFS/Hadoop configuration which is based on defaults
and overriding the one property {code}fs.defaultFS{code}. This behaviour is very limiting
for the use of HdfsResource since many other custom properties may be needed.
> Instead we should offer a way where users of the class can specify a directory with the
Hadoop configuration files. This could eventually also allow for a more flexibly style of
representing the HDFS path, namely without the hostname and port (which would simply be dynamically
loaded from the configuration file), like this:
> {code}
> hdfs:///path/to/file
> {code}
> How the configuration directory is provided should probably also be flexible. I would
suggest that we offer three ways:
> (1) a new constructor option to provide the directory, and
> (2) a system property that will be checked, and
> (3) an environmental variable that will be checked (I suggest HADOOP_CONF_DIR and YARN_CONF_DIR
to replicate common environmental variable names for this).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message