hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
Date Wed, 26 Aug 2009 05:42:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747790#action_12747790

Hemanth Yamijala commented on MAPREDUCE-893:

As explained above, we have this feature for the properties that the framework handles w.r.to
queues - ACLs and state.

We can draw the scope of this JIRA from what exists already.

- The existing framework for refresh of ACLs and state relies on a command *hadoop mradmin
- This command causes QueueManager to reload the configuration from the mapred-queues.xml
- If there's a syntactic or semantic error in reload, the refresh command fails with an exception
that is sent back to the hadoop mradmin command.
- Importantly, the existing configuration is untouched and the system is left in a consistent
- The UGI of the administrator who raised the refresh is logged to the JT log for audit purposes.

I believe all of these generic requirements will apply for the current JIRA as well.

The extension of scope is in the following manner:

- First the framework processes a reload of the configuration for properties it manages.
- If it passes, the framework will call the scheduler to refresh queue properties that are
managed by the scheduler. However, it will not commit its changes until this call succeeds.
- In MAPREDUCE-861, we suggested that schedulers which could define their properties as key-value
pairs can do so in the format we suggest [here|https://issues.apache.org/jira/browse/MAPREDUCE-861?focusedCommentId=12744910&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12744910].
(Look at the *properties* tag under queue). The framework can pass the list of properties
per queue to the scheduler, maybe as a Map of queue-name and properties.
- If the scheduler cannot process this information, it will throw an error (via an exception
or a return value) and discard the changes itself. The framework will likewise discard it
own changes and return error to the client with an appropriate message.
- It is possible that some of the properties may not be refresh-able. For e.g. we are not
going to handle new queues getting added or deleted. I think we should give a return value
back to the refreshQueues to indicate this.

Does this work ?

> Provide an ability to refresh queue configuration without restart.
> ------------------------------------------------------------------
>                 Key: MAPREDUCE-893
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-893
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Hemanth Yamijala
> While administering a cluster using multiple queues, administrators feel a need to refresh
queue properties on the fly without needing to restart the JobTracker. This is partially supported
for some properties such as queue ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is
to extend the facility to refresh other queue properties as well, including scheduler properties.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message