hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ying Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5520) [Capacity Scheduler] Change the logic for when to trigger user/group mappings to queues
Date Wed, 17 Aug 2016 03:11:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15423792#comment-15423792

Ying Zhang commented on YARN-5520:

Hi [~mshen], the ability of specifying another queue instead of the default mapping queue
sounds great.
Introducing a new parameter like yarn.scheduler.capacity.queue-mappings.disabled.queues can
do this. Alternatively, how about we extend the current queue mapping a little bit? Instead
of one queue per user or user group in the queue mapping, we could allow  admin to specify
one primary queue, and one or more secondary queues per user or user groups. When yarn.scheduler.capacity.queue-mappings-override.enable
is set to true, if user submits job without specifying a queue, use the primary mapping queue,
otherwise, use the specified queue as long as it is valid, it could be either the primary
queue or one of the secondary queues.

> [Capacity Scheduler] Change the logic for when to trigger user/group mappings to queues
> ---------------------------------------------------------------------------------------
>                 Key: YARN-5520
>                 URL: https://issues.apache.org/jira/browse/YARN-5520
>             Project: Hadoop YARN
>          Issue Type: Improvement
>    Affects Versions: 2.6.0, 2.7.0, 2.6.1
>            Reporter: Min Shen
> In YARN-2411, the feature in Capacity Scheduler to support user/group based mappings
to queues was introduced.
> In the original implementation, the configuration key {{yarn.scheduler.capacity.queue-mappings-override.enable}}
was added to control when to enable overriding user requested queues.
> However, even if this configuration is set to false, queue overriding could still happen
if the user didn't request for any specific queue or choose to simply submit his job to "default"
queue, according to the following if condition which triggers queue overriding:
> {code}
> if (queueName.equals(YarnConfiguration.DEFAULT_QUEUE_NAME)
>               || overrideWithQueueMappings)
> {code}
> This logic does not seem very reasonable, as there's no way to fully disable queue overriding
when mappings are configured inside capacity-scheduler.xml.
> In addition, in our environment, we have setup a few organization dedicated queues as
well as some "adhoc" queues. The organization dedicated queues have better resource guarantees
and we want to be able to route users to the corresponding organization queues. On the other
hand, the "adhoc" queues have less resource guarantees but everyone can use it to get some
opportunistic resources when the cluster is free.
> The current logic will also prevent this type of use cases as when you enable queue overriding,
users cannot use these "adhoc" queues any more. They will always be routed to the dedicated
organization queues.
> To address the above 2 issues, I propose to change the implementation so that:
> * Admin can fully disable queue overriding even if mappings are already configured.
> * Admin have finer grained control to cope queue overriding with the above mentioned
organization/adhoc queue setups.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message