hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-861) Modify queue configuration format and parsing to support a hierarchy of queues.
Date Fri, 14 Aug 2009 10:27:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743165#action_12743165
] 

Hemanth Yamijala commented on MAPREDUCE-861:
--------------------------------------------

Current state:
- Queue configuration lives in mapred-queues.xml. It contains the list of queues, their ACLs
and state. The pulling of these properties into a separate file happened in HADOOP-5396 and
HADOOP-5913 in Hadoop 0.21. Scheduler specific queue properties live in capacity-scheduler.xml.
They have the same format of property names and values as Hadoop configuration (like mapred-site.xml).
- For backwards compatibility, the queue configuration is also supported in mapred-site.xml,
but is deprecated.

The considerations are:
- To change the format of the file to be more friendly towards hierarchical queues. The file
format can be based on a well defined XML schema that we can discuss and agree upon, that
makes sense for nested queues. The other option for the file format was JSON. However, in
internal discussions, it was decided that it is best not to introduce a new format for managing
this information now. XML -> XML migration is less intrusive. Also, it is similar in format
to other configuration in Hadoop.
- We have an option to merge the contents of mapred-queues.xml and capacity-scheduler.xml.
It seems good to be able to do this because then the nested hierarchy of queues will be defined
in exactly one place and hence chances of users making mistakes in configuring the hierarchy
is lesser. Otherwise, the need for validation tools greatly increases.
- If we do agree to merge the contents, we may want to make scheduler specific properties
be treated as a black-box for the framework. An option could be to treat it as string contents.
And QueueManager could provide an API like getSchedulerProperties(queue), which returns a
string.
- For backwards compatibility, queue properties can still be defined in mapred-site.xml in
the old format, but they cannot define hierarchies of queues there.

Does this make sense overall ?


> Modify queue configuration format and parsing to support a hierarchy of queues.
> -------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-861
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-861
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>            Reporter: Hemanth Yamijala
>            Assignee: rahul k singh
>
> MAPREDUCE-853 proposes to introduce a hierarchy of queues into the Map/Reduce framework.
This JIRA is for defining changes to the configuration related to queues. 
> The current format for defining a queue and its properties is as follows: mapred.queue.<queue-name>.<property-name>.
For e.g. mapred.queue.<queue-name>.acl-submit-job. The reason for using this verbose
format was to be able to reuse the Configuration parser in Hadoop. However, administrators
currently using the queue configuration have already indicated a very strong desire for a
more manageable format. Since, this becomes more unwieldy with hierarchical queues, the time
may be good to introduce a new format for representing queue configuration.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message