hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matei Zaharia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-707) Provide a jobconf property for explicitly assigning a job to a pool
Date Wed, 04 Nov 2009 21:21:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773645#action_12773645

Matei Zaharia commented on MAPREDUCE-707:

Here are some comments on the patch:

# Instead of using the string "mapred.fairscheduler.pool" in multiple places in PoolManager,
make it a constant at the top of the file (something like EXPLICIT_POOL_PROPERTY).
# Add a comment to PoolManager.getPoolName to explain the logic (first look for the explicit
pool property, then for the property named by poolNameProperty, and finally default to DEFAULT_POOL_NAME).
# Add a unit test for PoolManager.getPoolName that tries each of those cases (explicit property
is set, no explicit property but poolNameProperty is used, or neither is used). Right now
your existing unit test checks that setPool works but there's no test that submits a job with
mapred.fairscheduler.pool directly.
# Instead of assertEquals(0,    scheduler.getPoolManager().getPoolName(job2).compareTo("poolA"))
you can probably use a version of assertEquals that works on strings.
# In the documentation, instead of saying "This property is ignored if mapred.fairscheduler.pool
is specified." for the poolnameproperty, it would be better to say that the poolnameproperty
is used only for jobs in which mapred.fairscheduler.pool is not explicitly set.

> Provide a jobconf property for explicitly assigning a job to a pool
> -------------------------------------------------------------------
>                 Key: MAPREDUCE-707
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-707
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: contrib/fair-share
>            Reporter: Matei Zaharia
>            Priority: Trivial
>         Attachments: MAPREDUCE-707-apache.patch
> A common use case of the fair scheduler is to have one pool per user, but then to define
some special pools for various production jobs, import jobs, etc. Therefore, it would be nice
if jobs went by default to the pool of the user who submitted them, but there was a setting
to explicitly place a job in another pool. Today, this can be achieved through a sort of trick
in the JobConf:
> {code}
> <property>
>   <name>mapred.fairscheduler.poolnameproperty</name>
>   <value>pool.name</value>
> </property>
> <property>
>   <name>pool.name</name>
>   <value>${user.name}</value>
> </property>
> {code}
> This JIRA proposes to add a property called mapred.fairscheduler.pool that allows a job
to be placed directly into a pool, avoiding the need for this trick.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message