From mapreduce-issues-return-5920-apmail-hadoop-mapreduce-issues-archive=hadoop.apache.org@hadoop.apache.org Tue Nov 03 08:10:23 2009 Return-Path: Delivered-To: apmail-hadoop-mapreduce-issues-archive@minotaur.apache.org Received: (qmail 27451 invoked from network); 3 Nov 2009 08:10:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 3 Nov 2009 08:10:22 -0000 Received: (qmail 81527 invoked by uid 500); 3 Nov 2009 08:10:22 -0000 Delivered-To: apmail-hadoop-mapreduce-issues-archive@hadoop.apache.org Received: (qmail 81472 invoked by uid 500); 3 Nov 2009 08:10:22 -0000 Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: mapreduce-issues@hadoop.apache.org Delivered-To: mailing list mapreduce-issues@hadoop.apache.org Received: (qmail 81461 invoked by uid 99); 3 Nov 2009 08:10:22 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 08:10:22 +0000 X-ASF-Spam-Status: No, hits=-10.5 required=5.0 tests=AWL,BAYES_00,RCVD_IN_DNSWL_HI X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2009 08:10:20 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 77E41234C052 for ; Tue, 3 Nov 2009 00:09:59 -0800 (PST) Message-ID: <513647598.1257235799476.JavaMail.jira@brutus> Date: Tue, 3 Nov 2009 08:09:59 +0000 (UTC) From: "Matei Zaharia (JIRA)" To: mapreduce-issues@hadoop.apache.org Subject: [jira] Commented: (MAPREDUCE-707) Provide a jobconf property for explicitly assigning a job to a pool In-Reply-To: <1603395329.1246815014830.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/MAPREDUCE-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772933#action_12772933 ] Matei Zaharia commented on MAPREDUCE-707: ----------------------------------------- I don't think that does what you want it to do, because it just sets poolNameProperty to a different value. What you want is for each *individual* job's pool to be determined based on which properties are in its JobConf (if it has mapred.fairscheduler.pool, use that; otherwise use whatever property poolNameProperty is set to; and if that doesn't exist in the JobConf either, use DEFAULT_POOL_NAME). The XML code I posted makes the job's JobConf implement this logic, by having the "pool" property be whatever the user sets it to if the user provides a setting, and making it default to the value of user.name otherwise. But there's no way to achieve this same behavior by just changing PoolManager.initialize to set poolNameProperty another way. Having thought about this some more, I think the most elegant implementation is to actually change PoolManger.getPoolName(JobInProgress job) to explicitly check whether the job's JobConf has the key "mapred.fairscheduler.pool" set, and if so, return that value; otherwise, it should return conf.get(poolNameProperty) as it does now. This should be a simple change to PoolManager.getPoolName. You will also need to change PoolManager.setPool() to set the "mapred.fairscheduler.pool" property rather than the poolNameProperty property on the job. Let me know if this makes sense. > Provide a jobconf property for explicitly assigning a job to a pool > ------------------------------------------------------------------- > > Key: MAPREDUCE-707 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-707 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share > Reporter: Matei Zaharia > Priority: Trivial > > A common use case of the fair scheduler is to have one pool per user, but then to define some special pools for various production jobs, import jobs, etc. Therefore, it would be nice if jobs went by default to the pool of the user who submitted them, but there was a setting to explicitly place a job in another pool. Today, this can be achieved through a sort of trick in the JobConf: > {code} > > mapred.fairscheduler.poolnameproperty > pool.name > > > pool.name > ${user.name} > > {code} > This JIRA proposes to add a property called mapred.fairscheduler.pool that allows a job to be placed directly into a pool, avoiding the need for this trick. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.