hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3579) Support property groups in Hadoop configuration
Date Tue, 17 Jun 2008 10:25:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605538#action_12605538
] 

Hemanth Yamijala commented on HADOOP-3579:
------------------------------------------

HADOOP-3479 asks for implementing configuration support for a Hadoop resource manager (HADOOP-3421).
Part of this configuration includes administrative queues of jobs. Each such queue would have
its own set of properties. It would be ideal to reuse the configuration of Hadoop, so that
administrators see a consistent experience in configuring Hadoop. From a development standpoint,
there would be consistency in the APIs, as well as reuse of code for various features.

To derive these benefits, Hadoop needs to support the following at a minimum:
- Grouping of related properties (for e.g. all properties related to a queue are grouped)
- Having multiple instances of a group (for e.g. there can be multiple instances of a queue)
- APIs for listing groups, their instances, and properties for a group's instance.

One proposal is to do the following:

- Add a <group> element that can appear at the same level as a <property> element
in the current Hadoop configuration.
- The group element can have the following format:
{code:xml}
<group>
  <name>hadoop.rm.queue</name>
  <instance>
    <name>research</name>
    <property>
      <name>hadoop.rm.queue.max-capacity</name>
      <value>100</value>
    </property>
  </instance>
</group>
{code}

Then we could add APIs such as:
{code}
Set<String> getGroups();
Set<String> getInstances(String groupname);
String get(String groupname, String instancename, String propertyname);
{code}

The behavior of the following features needs to be defined for groups:
- Loading configuration from multiple resources
- Final parameters
- Variable substitution

We could define that a property of a group's instance from a resource that is loaded latter
overrides an existing value, unless it is marked final. The granularity is at the level of
a (group, instance, property) tuple.

For variable substitution, we should define a new syntax to refer to variables in the instance
of a group, something like ${groupname[instance].propertyname}

The existing configuration API can be retained as is, because there can be properties that
do not belong to any group, as they exist today in hadoop-default and hadoop-site.

> Support property groups in Hadoop configuration
> -----------------------------------------------
>
>                 Key: HADOOP-3579
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3579
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: conf
>            Reporter: Hemanth Yamijala
>
> Hadoop configuration is currently a list of key and value pairs. There are some use-cases
to support configuring groups of related properties. There could also be multiple instances
of such groups. The issue is for adding support for such configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message