hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-707) Provide a jobconf property for explicitly assigning a job to a pool
Date Thu, 05 Nov 2009 00:55:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773733#action_12773733

Matei Zaharia commented on MAPREDUCE-707:

This looks pretty good, except that testPoolAssignment fails when I run the unit tests. I
think the problem is with job4, where you set "mapred.fairscheduler.poolnameproperty" in the
job's Configuration (jobConf2), not in the fair scheduler's configuration. You need to set
the poolNameProperty when you create the fair scheduler object. That's what the code used
to do with the POOL_PROPERTY string at the top, but you can't set the pool name property to
mapred.fairscheduler.pool, because that wouldn't be testing anything. I'd suggest leaving
the POOL_PROPERTY as "pool" and trying to set job4's pool through that.

Also, for sanity, in job1 (where you set mapred.fairscheduler.pool directly), you should say
the "pool" property to something other than poolA to make sure it isn't used.

Finally, two small nitpicks:

# In the test line with assertEquals(scheduler.getPoolManager().getPoolName(job2), "poolA"),
you should switch the two parameters (put "poolA" first); the first parameter is always supposed
to be the value expected.
# Regarding the comment on getPoolName - the pool name property used by default is "user.name",
not "project". I think I forgot to fix that comment a while back.

> Provide a jobconf property for explicitly assigning a job to a pool
> -------------------------------------------------------------------
>                 Key: MAPREDUCE-707
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-707
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>            Priority: Trivial
>         Attachments: MAPREDUCE-707-1-apache.patch, MAPREDUCE-707-apache.patch
> A common use case of the fair scheduler is to have one pool per user, but then to define
some special pools for various production jobs, import jobs, etc. Therefore, it would be nice
if jobs went by default to the pool of the user who submitted them, but there was a setting
to explicitly place a job in another pool. Today, this can be achieved through a sort of trick
in the JobConf:
> {code}
> <property>
>   <name>mapred.fairscheduler.poolnameproperty</name>
>   <value>pool.name</value>
> </property>
> <property>
>   <name>pool.name</name>
>   <value>${user.name}</value>
> </property>
> {code}
> This JIRA proposes to add a property called mapred.fairscheduler.pool that allows a job
to be placed directly into a pool, avoiding the need for this trick.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message