hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Wangda Tan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5949) Add pluggable configuration policy interface as a component of MutableCSConfigurationProvider
Date Wed, 03 May 2017 00:22:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994089#comment-15994089

Wangda Tan commented on YARN-5949:

Thanks [~jhung],

bq. This makes sense, should just be able to either grab the second-to-last component of the
key, or the component before "accessible-node-labels"/"ordering-policy". I think we can also
search for "root", if it is not there then assume it is a global config change.

I think we can handle it in this way:
There're a set of known queue paths like \{"root.queueA", "root.queueA.A1\}. For a given config
key to change, first we need to remove the common prefix ("yarn.scheduler.capacity."), do
the longest prefix match in the known queue paths.
- If we can find any non-empty common prefix, check the queue's accessibilities.
- If we cannot find, this is a global config, check admin permission.

This approach doesn't need to handle special options like "accessible-node-labels", and don't
need to use "root" to index starting of queue path, to me it is not a safe approach.

bq. As long as we access these queues via YarnScheduler#getQueueInfo, is this API still necessary?
When the scheduler is reinitialized and the next mutation comes in, it will check against
the queues from the most recent reinitialization.

We may have to call getQueueInfo for everytime when config mutation request comes in, the
frequency of mutation request should not super high, I think the stateless approach should
be fine.

bq. This is not implemented yet, but I was thinking of handling this in RMWebServices, there
are some cases that have not been handled (e.g. updating config for a queue which doesn't
exist shouldn't be allowed, right now it "succeeds" silently). So we can address these cases
in a separate jira.

Agree, it will add a never used option, it doesn't sound like a critical issue, we can handle
it in a separate JIRA. 

bq. Yes you're right, this is not handled yet, in fact there is still some handling we need
to do in RMWebServices for global configs, we can address this in a separate jira as well.

If non-trivial effort need to take for this, I'm OK to move it to a separate JIRA. This is
quite important to me. In fact, I think we should not assume any scheduler-specific configurations
inside RMWebServices (like add special logics to handle "yarn.scheduler.capacity.").


> Add pluggable configuration policy interface as a component of MutableCSConfigurationProvider
> ---------------------------------------------------------------------------------------------
>                 Key: YARN-5949
>                 URL: https://issues.apache.org/jira/browse/YARN-5949
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jonathan Hung
>            Assignee: Jonathan Hung
>         Attachments: YARN-5949-YARN-5734.001.patch, YARN-5949-YARN-5734.002.patch
> This will allow different policies to customize how/if configuration changes should be
applied (for example, a policy might restrict whether a configuration change by a certain
user is allowed). This will be enforced by the MutableCSConfigurationProvider.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message