hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9117) Config file reader / options classes for libhdfs++
Date Fri, 06 Nov 2015 21:32:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14994468#comment-14994468

Haohui Mai commented on HDFS-9117:

Thinking about the patch a little bit more, I believe that there is little value to follow
the original implementation in Java. The implementation looks over complicated. 

What is the minimal pieces of loading the configuration? Does the class need to address every
aspect of the concerns?

To me the core part of the patch should be parsing XML and populating the map of configuration.
Functionality like searching through the filesystem, environment variables, expanding the
configuration, etc., are all optional.

1. Note that the main motivation of the {{Options}} class is to make all the configurations
standalone and explicit. Users should be able to specify all configuration through the {{Options}}
object. Default values are fundamental parts of the contracts in the {{Options}} class. Filling
the configuration with the default values from the {{-*default.xml}} creates inconsistency
and bugs that are to be detected. The flip side is that {{Options}} can get out of dated if
someone changes the default value of the configuration, however it can be caught effectively
through adding a unit test.

2. Adding search paths and parsing them can be replaced by passing in a {{vector}} of path.
Parsing the environment variable is specific to to the compatibility layer of {{libhdfs}}.

3. Some of the functionality might be useful at later time. Since the code that uses that
functionality is yet-to-be-written, it is difficult to review and justify what is the appropriate
design and implementation. We can revisit some of these issues one the code that actually
uses the functionality is available.

I propose the following interfaces:

class Configuration {
  enum Priority {
   * Load configurations that are specified in XML format.
   * Each configuration file is associated with a priority.
   * A configuration with higher priority will overwrite the ones with lower priority.
  int ParseXMLConfiguration(const std::string &xml, Priority priority);

   * Get the value configuration, return empty if it's unspecified.
  template<class T>
  Optional<T> get(const std::string &key);

   * Get the value configuration, return the default value of configuration
   * if it's unspecified.
  template<class T>
  T getWithDefault(const std::string &key);


> Config file reader / options classes for libhdfs++
> --------------------------------------------------
>                 Key: HDFS-9117
>                 URL: https://issues.apache.org/jira/browse/HDFS-9117
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs-client
>    Affects Versions: HDFS-8707
>            Reporter: Bob Hansen
>            Assignee: Bob Hansen
>         Attachments: HDFS-9117.HDFS-8707.001.patch, HDFS-9117.HDFS-8707.002.patch, HDFS-9117.HDFS-8707.003.patch,
HDFS-9117.HDFS-8707.004.patch, HDFS-9117.HDFS-8707.005.patch, HDFS-9117.HDFS-8707.006.patch,
HDFS-9117.HDFS-8707.008.patch, HDFS-9117.HDFS-8707.009.patch, HDFS-9117.HDFS-8707.010.patch,
HDFS-9117.HDFS-8707.011.patch, HDFS-9117.HDFS-8707.012.patch, HDFS-9117.HDFS-9288.007.patch
> For environmental compatability with HDFS installations, libhdfs++ should be able to
read the configurations from Hadoop XML files and behave in line with the Java implementation.
> Most notably, machine names and ports should be readable from Hadoop XML configuration
> Similarly, an internal Options architecture for libhdfs++ should be developed to efficiently
transport the configuration information within the system.

This message was sent by Atlassian JIRA

View raw message