hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hemanth Yamijala (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-893) Provide an ability to refresh queue configuration without restart.
Date Wed, 26 Aug 2009 05:42:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747790#action_12747790
] 

Hemanth Yamijala commented on MAPREDUCE-893:
--------------------------------------------

As explained above, we have this feature for the properties that the framework handles w.r.to
queues - ACLs and state.

We can draw the scope of this JIRA from what exists already.

- The existing framework for refresh of ACLs and state relies on a command *hadoop mradmin
-refreshQueues*. 
- This command causes QueueManager to reload the configuration from the mapred-queues.xml
file.
- If there's a syntactic or semantic error in reload, the refresh command fails with an exception
that is sent back to the hadoop mradmin command.
- Importantly, the existing configuration is untouched and the system is left in a consistent
state
- The UGI of the administrator who raised the refresh is logged to the JT log for audit purposes.

I believe all of these generic requirements will apply for the current JIRA as well.

The extension of scope is in the following manner:

- First the framework processes a reload of the configuration for properties it manages.
- If it passes, the framework will call the scheduler to refresh queue properties that are
managed by the scheduler. However, it will not commit its changes until this call succeeds.
- In MAPREDUCE-861, we suggested that schedulers which could define their properties as key-value
pairs can do so in the format we suggest [here|https://issues.apache.org/jira/browse/MAPREDUCE-861?focusedCommentId=12744910&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12744910].
(Look at the *properties* tag under queue). The framework can pass the list of properties
per queue to the scheduler, maybe as a Map of queue-name and properties.
- If the scheduler cannot process this information, it will throw an error (via an exception
or a return value) and discard the changes itself. The framework will likewise discard it
own changes and return error to the client with an appropriate message.
- It is possible that some of the properties may not be refresh-able. For e.g. we are not
going to handle new queues getting added or deleted. I think we should give a return value
back to the refreshQueues to indicate this.

Does this work ?

> Provide an ability to refresh queue configuration without restart.
> ------------------------------------------------------------------
>
>                 Key: MAPREDUCE-893
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-893
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Hemanth Yamijala
>
> While administering a cluster using multiple queues, administrators feel a need to refresh
queue properties on the fly without needing to restart the JobTracker. This is partially supported
for some properties such as queue ACLs (HADOOP-5396) and state (HADOOP-5913). The idea is
to extend the facility to refresh other queue properties as well, including scheduler properties.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message