From yarn-issues-return-33831-apmail-hadoop-yarn-issues-archive=hadoop.apache.org@hadoop.apache.org Sun Aug 17 18:15:20 2014 Return-Path: X-Original-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-yarn-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6B264114E7 for ; Sun, 17 Aug 2014 18:15:20 +0000 (UTC) Received: (qmail 57524 invoked by uid 500); 17 Aug 2014 18:15:20 -0000 Delivered-To: apmail-hadoop-yarn-issues-archive@hadoop.apache.org Received: (qmail 57482 invoked by uid 500); 17 Aug 2014 18:15:20 -0000 Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: yarn-issues@hadoop.apache.org Delivered-To: mailing list yarn-issues@hadoop.apache.org Received: (qmail 57469 invoked by uid 99); 17 Aug 2014 18:15:20 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 17 Aug 2014 18:15:20 +0000 Date: Sun, 17 Aug 2014 18:15:20 +0000 (UTC) From: "Jian He (JIRA)" To: yarn-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (YARN-2411) [Capacity Scheduler] support simple user and group mappings to queues MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/YARN-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14100029#comment-14100029 ] Jian He commented on YARN-2411: ------------------------------- test failures should be unrelated to the patch. resubmit same patch with one more assertion of RMApp.getQueue in the test. > [Capacity Scheduler] support simple user and group mappings to queues > --------------------------------------------------------------------- > > Key: YARN-2411 > URL: https://issues.apache.org/jira/browse/YARN-2411 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler > Reporter: Ram Venkatesh > Assignee: Ram Venkatesh > Attachments: YARN-2411-2.patch, YARN-2411.1.patch, YARN-2411.3.patch, YARN-2411.4.patch, YARN-2411.5.patch > > > YARN-2257 has a proposal to extend and share the queue placement rules for the fair scheduler and the capacity scheduler. This is a good long term solution to streamline queue placement of both schedulers but it has core infra work that has to happen first and might require changes to current features in all schedulers along with corresponding configuration changes, if any. > I would like to propose a change with a smaller scope in the capacity scheduler that addresses the core use cases for implicitly mapping jobs that have the default queue or no queue specified to specific queues based on the submitting user and user groups. It will be useful in a number of real-world scenarios and can be migrated over to the unified scheme when YARN-2257 becomes available. > The proposal is to add two new configuration options: > yarn.scheduler.capacity.queue-mappings-override.enable > A boolean that controls if user-specified queues can be overridden by the mapping, default is false. > and, > yarn.scheduler.capacity.queue-mappings > A string that specifies a list of mappings in the following format (default is "" which is the same as no mapping) > ::[,::]* > map_specifier := user (u) | group (g) > source_attribute := user | group | %user > queue_name := the name of the mapped queue | %user | %primary_group > The mappings will be evaluated left to right, and the first valid mapping will be used. If the mapped queue does not exist, or the current user does not have permissions to submit jobs to the mapped queue, the submission will fail. > Example usages: > 1. user1 is mapped to queue1, group1 is mapped to queue2 > u:user1:queue1,g:group1:queue2 > 2. To map users to queues with the same name as the user: > u:%user:%user > I am happy to volunteer to take this up. -- This message was sent by Atlassian JIRA (v6.2#6252)