hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Chansler (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4768) Dynamic Priority Scheduler that allows queue shares to be controlled dynamically by a currency
Date Thu, 08 Oct 2009 04:17:31 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Chansler updated HADOOP-4768:
------------------------------------

    Release Note: New contribution Dynamic Scheduler implements dynamic priorities with a
currency model. Usage instructions are in the Jira item.  (was: This package implements dynamic
priority scheduling for MapReduce jobs.

Overview
--------
The purpose of this scheduler is to allow users to increase and decrease
their queue priorities continuosly to meet the requirements of their
current workloads. The scheduler is aware of the current demand and makes
it more expensive to boost the priority under peak usage times. Thus
users who move their workload to low usage times are rewarded with
discounts. Priorities can only be boosted within a limited quota.
All users are given a quota or a budget which is deducted periodically
in configurable accounting intervals. How much of the budget is
deducted is determined by a per-user spending rate, which may
be modified at any time directly by the user. The cluster slots
share allocated to a particular user is computed as that users
spending rate over the sum of all spending rates in the same accounting
period.

Configuration
-------------
This scheduler comprises two components, an accounting or resource allocation part that
manages and bills for queue shares, and a scheduler that
enforces the queue shares in the form of map and reduce slots of running jobs.

Hadoop Configuration (e.g. hadoop-site.xml):
mapred.jobtracker.taskScheduler
    This needs to be set to
    org.apache.hadoop.mapred.DynamicPriorityScheduler
    to use the dynamic scheduler.
Scheduler Configuration:
mapred.dynamic-scheduler.scheduler
    The Java path of the MapReduce scheduler that should
    enforce the allocated shares.
    Has been tested with (which is the default):
    org.apache.hadoop.mapred.PriorityScheduler
mapred.priority-scheduler.acl-file
    Full path of ACL with syntax:
      <user> <role> <secret key>
    separated by line feeds
mapred.dynamic-scheduler.budget-file
    The full OS path of the file from which the
    budgets are read and stored. The syntax of this file is:
    <queue name> <budget> <spending rate>
    separated by newlines where budget can be specified
    as a Java float. The file should not be edited
    directly, if the server is running, but through the
    servlet API to ensure proper synchronization.

mapred.dynamic-scheduler.alloc-interval
    Allocation interval, when the scheduler rereads the
    spending rates and recalculates the cluster shares.
    Specified as seconds between allocations.
    Default is 20 seconds.

)

Editorial pass over all release notes prior to publication of 0.21.

Usage instructions:


Overview
--------
The purpose of this scheduler is to allow users to increase and decrease
their queue priorities continuosly to meet the requirements of their
current workloads. The scheduler is aware of the current demand and makes
it more expensive to boost the priority under peak usage times. Thus
users who move their workload to low usage times are rewarded with
discounts. Priorities can only be boosted within a limited quota.
All users are given a quota or a budget which is deducted periodically
in configurable accounting intervals. How much of the budget is
deducted is determined by a per-user spending rate, which may
be modified at any time directly by the user. The cluster slots
share allocated to a particular user is computed as that users
spending rate over the sum of all spending rates in the same accounting
period.

Configuration
-------------
This scheduler comprises two components, an accounting or resource allocation part that
manages and bills for queue shares, and a scheduler that
enforces the queue shares in the form of map and reduce slots of running jobs.

Hadoop Configuration (e.g. hadoop-site.xml):
mapred.jobtracker.taskScheduler
    This needs to be set to
    org.apache.hadoop.mapred.Editorial pass over all release notes prior to publication of
0.21.

    to use the dynamic scheduler.
Scheduler Configuration:
mapred.dynamic-scheduler.scheduler
    The Java path of the MapReduce scheduler that should
    enforce the allocated shares.
    Has been tested with (which is the default):
    org.apache.hadoop.mapred.PriorityScheduler
mapred.priority-scheduler.acl-file
    Full path of ACL with syntax:
      <user> <role> <secret key>
    separated by line feeds
mapred.dynamic-scheduler.budget-file
    The full OS path of the file from which the
    budgets are read and stored. The syntax of this file is:
    <queue name> <budget> <spending rate>
    separated by newlines where budget can be specified
    as a Java float. The file should not be edited
    directly, if the server is running, but through the
    servlet API to ensure proper synchronization.

mapred.dynamic-scheduler.alloc-interval
    Allocation interval, when the scheduler rereads the
    spending rates and recalculates the cluster shares.
    Specified as seconds between allocations.
    Default is 20 seconds.



> Dynamic Priority Scheduler that allows queue shares to be controlled dynamically by a
currency
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4768
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4768
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.21.0
>            Reporter: Thomas Sandholm
>            Assignee: Thomas Sandholm
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-4768-2.patch, HADOOP-4768-3.patch, HADOOP-4768-4.patch, HADOOP-4768-5.patch,
HADOOP-4768-6.patch, HADOOP-4768-capacity-scheduler.patch, HADOOP-4768-dynamic-scheduler.patch,
HADOOP-4768-fairshare.patch, HADOOP-4768.patch
>
>
> Dynamic (economic) priority scheduler based on work presented at the Hadoop User Group
meeting in Santa Clara in September and the HadoopCamp in New Orleans in November 2008.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message