hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Esther Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6686) Support for adding and removing queue mappings
Date Sat, 26 Aug 2017 22:23:02 GMT

    [ https://issues.apache.org/jira/browse/YARN-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142952#comment-16142952
] 

Esther Zheng commented on YARN-6686:
------------------------------------

[~jhung] One of my concerns regarding the current proposal is the order of the params. 

Here is an example: 

Initial value: 
{code:java}
u:user1:queue1,u:%user:%user
{code}

{code:java}
addParams: u:user1:queue2
removeParams: u:user1:queue1
{code}

Actual value: 
{code:java}
u:%user:%user,u:user1:queue2
{code}

The current implementation adds the params to the end of the existing mappings. Note that
the queue-mapping configuration is evaluated from left to right. In this example, the user
'user1' will be mapped to the queue 'user1' because 'u:%user:%user' is listed before 'u:user1:queue2'.


However, the administrator may actually want to map 'user1' to 'queue2' (see below), i.e.,
'u:user1:queue2' is placed before 'u:%user:%user'. 

Expected value:
{code:java}
u:user1:queue2,u:%user:%user
{code}


To achieve the expected value, the administrator may use: 
{code:java}
addParams: u:user1:queue2,u:%user:%user
removeParams: u:user1:queue1,u:%user:%user
{code}

Such usage may be inconvenient. Furthermore, if multiple global-updates occur simultaneously,
the order still may not be the same as expected. 

Is this a valid concern?

cc [~wangda]

> Support for adding and removing queue mappings
> ----------------------------------------------
>
>                 Key: YARN-6686
>                 URL: https://issues.apache.org/jira/browse/YARN-6686
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jonathan Hung
>            Assignee: Jonathan Hung
>         Attachments: YARN-6686-YARN-5734.001.patch
>
>
> Right now capacity scheduler uses UserGroupMappingPlacementRule to determine queue mappings.
This rule stores mappings in {{yarn.scheduler.capacity.queue-mappings}}. For users with a
large number of mappings, adding or removing queue mappings becomes infeasible.
> Need to come up with a way to add/remove individual mappings, for any/all different configured
placement rules.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message